2024-03-30T13:36:11.725 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T13:36:11.726 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126 --description orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126/orig.config.yaml 2024-03-30T13:36:11.754 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T13:36:11.841 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126 branch: squid description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7630126' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 5af0400050a123dc31f105425174f106e057276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8777 sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb sleep_before_teardown: 0 subset: 43/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 5af0400050a123dc31f105425174f106e057276a targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmVzuHzB7tOyXUtvJQhmRnarhqnL7kpm8q5+RZnh9egEBaAopWK4pdCNeD8FPLU9PKFuTLSirtTGHU3PSSONdA= smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFIO3nS3DhsYqCK/Ntvgz3OpZ5cEeUTG/bEzhm96Uc4sb/TYQEwp5hc8N1hlG3RHk8vVqMdeRnIzHFb4IhAM9pk= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T13:36:11.841 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa; will attempt to use it 2024-03-30T13:36:11.842 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa/tasks 2024-03-30T13:36:11.842 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T13:36:11.844 INFO:teuthology.task.internal:Checking packages... 2024-03-30T13:36:11.867 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb' 2024-03-30T13:36:11.867 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T13:36:11.867 INFO:teuthology.packaging:ref: None 2024-03-30T13:36:11.867 INFO:teuthology.packaging:tag: None 2024-03-30T13:36:11.867 INFO:teuthology.packaging:branch: squid 2024-03-30T13:36:11.867 INFO:teuthology.packaging:sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:36:11.867 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-30T13:36:12.049 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1973.g5af04000 2024-03-30T13:36:12.051 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T13:36:12.060 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T13:36:12.060 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T13:36:12.078 INFO:teuthology.task.internal:Saving configuration 2024-03-30T13:36:12.088 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T13:36:12.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T13:36:12.125 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:31:26.845630', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmVzuHzB7tOyXUtvJQhmRnarhqnL7kpm8q5+RZnh9egEBaAopWK4pdCNeD8FPLU9PKFuTLSirtTGHU3PSSONdA='} 2024-03-30T13:36:12.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:31:26.847078', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFIO3nS3DhsYqCK/Ntvgz3OpZ5cEeUTG/bEzhm96Uc4sb/TYQEwp5hc8N1hlG3RHk8vVqMdeRnIzHFb4IhAM9pk='} 2024-03-30T13:36:12.149 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T13:36:12.185 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-30T13:36:12.185 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-30T13:36:12.186 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T13:36:12.262 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5e08313f70>, signals=[15]) 2024-03-30T13:36:12.262 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T13:36:12.269 INFO:teuthology.task.internal:Opening connections... 2024-03-30T13:36:12.269 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-30T13:36:12.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:36:12.361 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-03-30T13:36:12.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:36:12.448 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T13:36:12.460 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-30T13:36:12.476 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-30T13:36:12.476 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-03-30T13:36:12.532 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-03-30T13:36:12.533 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T13:36:12.533 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-03-30T13:36:12.533 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T13:36:12.533 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T13:36:12.533 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T13:36:12.533 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-30T13:36:12.561 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-03-30T13:36:12.579 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-03-30T13:36:12.579 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="9" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="9" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el9" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-03-30T13:36:12.636 INFO:teuthology.orchestra.run.smithi138.stdout:LOGO="fedora-logo-icon" 2024-03-30T13:36:12.637 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T13:36:12.637 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2024-03-30T13:36:12.637 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T13:36:12.637 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T13:36:12.637 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T13:36:12.637 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-03-30T13:36:12.661 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T13:36:12.673 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T13:36:12.679 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T13:36:12.679 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-30T13:36:12.682 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-03-30T13:36:12.697 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T13:36:12.704 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T13:36:12.704 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-30T13:36:12.738 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-03-30T13:36:12.840 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T13:36:12.893 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T13:36:12.909 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T13:36:12.909 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T13:36:12.910 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T13:36:12.910 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T13:36:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:36:12.926 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-30T13:36:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:36:12.927 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-03-30T13:36:12.983 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-361.el9.x86_64 2024-03-30T13:36:12.983 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-361.el9.x86_64 2024-03-30T13:36:12.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-03-30T13:36:12.984 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-361.el9.x86_64 2024-03-30T13:36:12.985 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.14.0-361.el9.x86_64 2024-03-30T13:36:12.985 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum install -y kernel 2024-03-30T13:36:15.119 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-30T13:36:15.310 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-30T13:36:16.528 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-30T13:36:16.710 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-30T13:36:17.927 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.0 MB 00:00 2024-03-30T13:36:23.402 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-30T13:36:27.530 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-30T13:36:28.886 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-30T13:36:33.049 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 5.1 MB/s | 21 MB 00:04 2024-03-30T13:36:38.021 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-30T13:36:40.228 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T13:36:40.228 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T13:36:40.324 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:36:40.334 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout:Install 4 Packages 2024-03-30T13:36:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:36:40.336 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 76 M 2024-03-30T13:36:40.336 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 123 M 2024-03-30T13:36:40.336 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:36:40.514 INFO:teuthology.orchestra.run.smithi138.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-30T13:36:41.297 INFO:teuthology.orchestra.run.smithi138.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-30T13:36:42.297 INFO:teuthology.orchestra.run.smithi138.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-03-30T13:36:43.797 INFO:teuthology.orchestra.run.smithi138.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 8.4 MB/s | 28 MB 00:03 2024-03-30T13:36:43.801 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:36:43.801 INFO:teuthology.orchestra.run.smithi138.stdout:Total 22 MB/s | 76 MB 00:03 2024-03-30T13:36:43.801 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:36:44.172 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:36:44.172 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:36:45.017 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:36:45.018 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:36:47.312 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:36:48.156 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:36:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:36:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:36:51.179 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:36:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:36:55.928 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:37:00.645 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:37:29.909 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:37:29.910 INFO:teuthology.orchestra.run.smithi138.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T13:37:29.910 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:37:29.953 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:37:30.343 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:37:30.344 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:37:30.344 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:37:30.344 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:37:30.747 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:37:31.081 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:37:31.190 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:37:31.190 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:37:31.190 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T13:37:31.190 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T13:37:31.191 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T13:37:31.191 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-30T13:37:31.191 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T13:37:31.191 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:37:31.270 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:37:31.271 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:37:31.271 DEBUG:teuthology.orchestra.run.smithi138:> sudo rpm -qi grub2-tools 2024-03-30T13:37:31.305 INFO:teuthology.orchestra.run.smithi138.stdout:Name : grub2-tools 2024-03-30T13:37:31.305 INFO:teuthology.orchestra.run.smithi138.stdout:Epoch : 1 2024-03-30T13:37:31.305 INFO:teuthology.orchestra.run.smithi138.stdout:Version : 2.06 2024-03-30T13:37:31.305 INFO:teuthology.orchestra.run.smithi138.stdout:Release : 68.el9 2024-03-30T13:37:31.305 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: x86_64 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Group : Unspecified 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Size : 8264254 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:License : GPLv3+ 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Packager : builder@centos.org 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Vendor : CentOS 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Summary : Support tools for GRUB. 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout:Description : 2024-03-30T13:37:31.306 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:37:31.307 INFO:teuthology.orchestra.run.smithi138.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T13:37:31.307 INFO:teuthology.orchestra.run.smithi138.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T13:37:31.307 INFO:teuthology.orchestra.run.smithi138.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T13:37:31.307 INFO:teuthology.orchestra.run.smithi138.stdout:hardware devices. 2024-03-30T13:37:31.307 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:37:31.307 INFO:teuthology.orchestra.run.smithi138.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T13:37:31.308 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T13:37:31.308 INFO:teuthology.task.kernel:Updating grub on smithi138 to boot 5.14.0-434.el9.x86_64 2024-03-30T13:37:31.308 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T13:37:31.892 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2024-03-30T13:37:33.318 INFO:teuthology.orchestra.run.smithi138.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T13:37:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:done 2024-03-30T13:37:33.359 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-03-30T13:37:33.374 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.FwUhmnFMOW 2024-03-30T13:37:33.374 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FwUhmnFMOW 2024-03-30T13:37:33.443 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.FwUhmnFMOW 2024-03-30T13:37:33.579 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.FwUhmnFMOW is 6KB 2024-03-30T13:37:33.628 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.FwUhmnFMOW 2024-03-30T13:37:33.642 DEBUG:teuthology.orchestra.run.smithi138:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T13:37:33.706 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T13:37:33.707 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T13:37:33.707 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T13:37:33.707 INFO:teuthology.orchestra.run.smithi138.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T13:37:33.708 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T13:37:33.862 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2024-03-30T13:37:49.776 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 89 kB/s | 8.0 MB 01:32 2024-03-30T13:37:53.085 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 19 MB 00:02 2024-03-30T13:37:57.102 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-30T13:37:59.036 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-30T13:37:59.863 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-30T13:38:03.865 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-30T13:38:03.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-30T13:38:03.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:38:04.805 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-30T13:38:07.011 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T13:38:07.011 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T13:38:07.104 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T13:38:07.117 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T13:38:07.118 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T13:38:07.118 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:38:07.118 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:38:07.118 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:38:07.118 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-03-30T13:38:07.118 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:38:07.118 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 76 M 2024-03-30T13:38:07.119 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 123 M 2024-03-30T13:38:07.119 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:38:07.331 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 902 kB/s | 192 kB 00:00 2024-03-30T13:38:08.165 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-30T13:38:08.815 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-30T13:38:09.940 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 11 MB/s | 28 MB 00:02 2024-03-30T13:38:09.943 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:38:09.943 INFO:teuthology.orchestra.run.smithi102.stdout:Total 27 MB/s | 76 MB 00:02 2024-03-30T13:38:09.944 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:38:10.306 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:38:10.306 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:38:11.147 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:38:11.148 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:38:13.460 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:38:14.118 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:38:14.207 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:38:16.977 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:38:17.120 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:38:21.840 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:38:21.886 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:38:22.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-30T13:38:26.693 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:38:31.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-30T13:38:31.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:38:34.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-30T13:38:46.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-30T13:38:46.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:38:56.118 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:38:56.118 INFO:teuthology.orchestra.run.smithi102.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T13:38:56.118 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:38:56.160 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:38:56.570 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:38:56.570 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T13:38:56.570 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T13:38:56.570 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T13:38:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:38:57.024 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:38:57.373 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:38:57.509 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:38:57.510 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:38:57.510 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T13:38:57.510 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T13:38:57.510 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T13:38:57.510 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-30T13:38:57.510 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T13:38:57.510 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T13:38:57.589 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T13:38:57.590 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T13:38:57.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-03-30T13:38:57.623 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-03-30T13:38:57.623 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-03-30T13:38:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:38:57.625 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T13:38:57.626 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T13:38:57.626 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 5.14.0-434.el9.x86_64 2024-03-30T13:38:57.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T13:38:58.222 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-30T13:38:59.661 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T13:38:59.689 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-30T13:38:59.691 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-30T13:38:59.706 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.9Uts88QBWY 2024-03-30T13:38:59.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9Uts88QBWY 2024-03-30T13:38:59.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.9Uts88QBWY 2024-03-30T13:38:59.910 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.9Uts88QBWY is 6KB 2024-03-30T13:38:59.960 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.9Uts88QBWY 2024-03-30T13:38:59.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T13:39:00.037 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T13:39:00.038 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T13:39:00.038 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T13:39:00.038 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T13:39:00.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T13:39:00.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-03-30T13:39:18.240 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-03-30T13:39:18.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-30T13:39:18.625 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-30T13:39:18.625 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-03-30T13:39:18.639 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-434.el9.x86_64 2024-03-30T13:39:18.639 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T13:39:18.639 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T13:39:18.639 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T13:39:19.640 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-30T13:39:19.640 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T13:39:19.707 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2024-03-30T13:39:19.743 DEBUG:teuthology.parallel:result is None 2024-03-30T13:39:30.206 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-30T13:39:30.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-30T13:39:30.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:39:48.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-30T13:39:57.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-30T13:39:57.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:40:57.643 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T13:41:09.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-30T13:41:09.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:41:10.033 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-30T13:41:10.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-30T13:41:10.421 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-30T13:41:10.421 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-30T13:41:10.436 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-434.el9.x86_64 2024-03-30T13:41:10.436 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T13:41:10.436 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T13:41:10.436 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T13:41:11.437 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-30T13:41:11.437 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T13:41:11.520 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-03-30T13:41:11.559 DEBUG:teuthology.parallel:result is None 2024-03-30T13:41:11.559 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T13:41:11.566 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T13:41:11.567 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T13:41:11.569 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T13:41:11.588 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T13:41:11.595 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T13:41:11.605 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T13:41:11.605 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T13:41:11.626 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T13:41:11.660 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T13:41:11.667 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T13:41:11.667 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T13:41:11.691 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T13:41:11.724 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:41:11.735 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:41:11.739 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:41:11.749 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T13:41:11.750 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T13:41:11.757 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T13:41:11.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T13:41:11.783 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T13:41:11.819 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T13:41:11.837 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T13:41:11.837 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T13:41:11.853 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T13:41:11.874 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:41:11.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:41:11.998 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:41:12.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:41:12.124 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:41:12.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T13:41:12.192 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:41:12.236 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T13:41:12.294 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:41:12.379 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T13:41:12.425 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:41:12.425 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T13:41:12.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-30T13:41:12.497 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-03-30T13:41:12.546 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T13:41:12.589 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T13:41:12.983 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T13:41:12.991 INFO:teuthology.task.internal:Starting timer... 2024-03-30T13:41:12.992 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T13:41:13.001 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T13:41:13.009 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-30T13:41:13.009 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-03-30T13:41:13.055 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-03-30T13:41:13.057 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2024-03-30T13:41:13.116 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2024-03-30T13:41:13.118 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T13:41:13.118 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-03-30T13:41:13.150 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-03-30T13:41:13.151 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2024-03-30T13:41:13.170 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2024-03-30T13:41:13.170 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi138.front.sepia.ceph.com': 'permissive'} 2024-03-30T13:41:13.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T13:41:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:41:13.219 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T13:41:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:41:13.248 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T13:41:13.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-03-30T13:41:13.294 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2024-03-30T13:41:13.329 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T13:41:13.339 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T13:41:13.356 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T13:41:13.357 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T13:46:03.745 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2024-03-30T13:46:03.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-30T13:46:03.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:46:03.840 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-30T13:46:03.913 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-30T13:46:03.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-30T13:46:03.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T13:46:04.020 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-03-30T13:46:04.098 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-30T13:46:04.099 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T13:46:04.110 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T13:46:04.110 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T13:46:04.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T13:46:04.113 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T13:46:04.113 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T13:46:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T13:46:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T13:46:04.170 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T13:46:04.187 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-03-30T13:46:04.188 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T13:46:04.201 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-03-30T13:46:04.217 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T13:46:04.218 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2024-03-30T13:46:04.231 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2024-03-30T13:46:04.233 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T13:46:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T13:46:04.264 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T13:46:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-30T13:46:04.286 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T13:46:04.286 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-30T13:46:04.286 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.286 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.286 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.287 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.317 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-03-30T13:46:04.321 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T13:46:04.321 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-03-30T13:46:04.322 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.322 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.322 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.322 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T13:46:04.322 INFO:teuthology.run_tasks:Running task install... 2024-03-30T13:46:04.330 DEBUG:teuthology.task.install:project ceph 2024-03-30T13:46:04.330 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'}, 'extra_packages': ['cephadm']} 2024-03-30T13:46:04.331 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'} 2024-03-30T13:46:04.331 INFO:teuthology.task.install:Using flavor: default 2024-03-30T13:46:04.336 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T13:46:04.337 INFO:teuthology.task.install:extra packages: [] 2024-03-30T13:46:04.337 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'tag': None, 'wait_for_package': False} 2024-03-30T13:46:04.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:46:04.340 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'tag': None, 'wait_for_package': False} 2024-03-30T13:46:04.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:46:04.506 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/ 2024-03-30T13:46:04.506 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1967.g4e50bbcd 2024-03-30T13:46:04.585 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/ 2024-03-30T13:46:04.585 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1967.g4e50bbcd 2024-03-30T13:46:04.704 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T13:46:04.704 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:46:04.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T13:46:04.715 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T13:46:04.715 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:46:04.715 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T13:46:04.732 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T13:46:04.732 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T13:46:04.743 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T13:46:04.744 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T13:46:04.800 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T13:46:04.811 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T13:46:04.880 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T13:46:04.892 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T13:46:04.946 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-03-30T13:46:04.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-03-30T13:46:04.958 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2024-03-30T13:46:04.959 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2024-03-30T13:46:05.190 INFO:teuthology.orchestra.run.smithi102.stdout:74 files removed 2024-03-30T13:46:05.207 INFO:teuthology.orchestra.run.smithi138.stdout:74 files removed 2024-03-30T13:46:05.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2024-03-30T13:46:05.235 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw 2024-03-30T13:46:05.987 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 176 kB/s | 83 kB 00:00 2024-03-30T13:46:06.024 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 170 kB/s | 83 kB 00:00 2024-03-30T13:46:06.687 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 18 kB/s | 11 kB 00:00 2024-03-30T13:46:06.741 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 17 kB/s | 11 kB 00:00 2024-03-30T13:46:07.067 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-30T13:46:07.205 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-30T13:46:07.915 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-30T13:46:08.362 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - AppStream 5.3 MB/s | 6.7 MB 00:01 2024-03-30T13:46:09.682 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-30T13:46:09.820 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-30T13:46:10.540 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-30T13:46:10.694 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-30T13:46:13.708 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 19 MB 00:02 2024-03-30T13:46:15.794 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-30T13:46:18.094 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.1 MB 00:00 2024-03-30T13:46:19.459 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-30T13:46:39.726 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-30T13:46:44.735 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-30T13:46:47.320 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:46:47.332 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 9.8 M 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.4 M 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.3 M 2024-03-30T13:46:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 21 M 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 25 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 745 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 501 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.1 M 2024-03-30T13:46:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 45 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 131 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 161 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 321 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 298 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 99 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-30T13:46:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout:Install 30 Packages 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2024-03-30T13:46:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:46:47.337 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 59 M 2024-03-30T13:46:47.337 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:46:48.360 INFO:teuthology.orchestra.run.smithi102.stdout:(1/32): ceph-base-19.0.0-1967.g4e50bbcd.el9.x86 6.7 MB/s | 5.3 MB 00:00 2024-03-30T13:46:48.410 INFO:teuthology.orchestra.run.smithi102.stdout:(2/32): ceph-selinux-19.0.0-1967.g4e50bbcd.el9. 507 kB/s | 25 kB 00:00 2024-03-30T13:46:48.511 INFO:teuthology.orchestra.run.smithi102.stdout:(3/32): libcephfs2-19.0.0-1967.g4e50bbcd.el9.x8 7.3 MB/s | 745 kB 00:00 2024-03-30T13:46:48.611 INFO:teuthology.orchestra.run.smithi102.stdout:(4/32): libradosstriper1-19.0.0-1967.g4e50bbcd. 4.9 MB/s | 501 kB 00:00 2024-03-30T13:46:48.745 INFO:teuthology.orchestra.run.smithi102.stdout:(5/32): ceph-radosgw-19.0.0-1967.g4e50bbcd.el9. 8.2 MB/s | 9.8 MB 00:01 2024-03-30T13:46:48.803 INFO:teuthology.orchestra.run.smithi102.stdout:(6/32): python3-ceph-argparse-19.0.0-1967.g4e50 776 kB/s | 45 kB 00:00 2024-03-30T13:46:48.870 INFO:teuthology.orchestra.run.smithi102.stdout:(7/32): python3-ceph-common-19.0.0-1967.g4e50bb 1.9 MB/s | 131 kB 00:00 2024-03-30T13:46:48.929 INFO:teuthology.orchestra.run.smithi102.stdout:(8/32): python3-cephfs-19.0.0-1967.g4e50bbcd.el 2.7 MB/s | 161 kB 00:00 2024-03-30T13:46:48.996 INFO:teuthology.orchestra.run.smithi102.stdout:(9/32): python3-rados-19.0.0-1967.g4e50bbcd.el9 4.7 MB/s | 321 kB 00:00 2024-03-30T13:46:49.071 INFO:teuthology.orchestra.run.smithi102.stdout:(10/32): librgw2-19.0.0-1967.g4e50bbcd.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-03-30T13:46:49.098 INFO:teuthology.orchestra.run.smithi102.stdout:(11/32): python3-rbd-19.0.0-1967.g4e50bbcd.el9. 2.9 MB/s | 298 kB 00:00 2024-03-30T13:46:49.121 INFO:teuthology.orchestra.run.smithi102.stdout:(12/32): python3-rgw-19.0.0-1967.g4e50bbcd.el9. 1.9 MB/s | 99 kB 00:00 2024-03-30T13:46:49.272 INFO:teuthology.orchestra.run.smithi102.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 604 kB/s | 106 kB 00:00 2024-03-30T13:46:49.299 INFO:teuthology.orchestra.run.smithi102.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 908 kB/s | 159 kB 00:00 2024-03-30T13:46:49.314 INFO:teuthology.orchestra.run.smithi102.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-30T13:46:49.389 INFO:teuthology.orchestra.run.smithi102.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-30T13:46:49.415 INFO:teuthology.orchestra.run.smithi102.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.7 MB/s | 282 kB 00:00 2024-03-30T13:46:49.442 INFO:teuthology.orchestra.run.smithi102.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-30T13:46:49.465 INFO:teuthology.orchestra.run.smithi102.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-30T13:46:49.482 INFO:teuthology.orchestra.run.smithi102.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 835 kB/s | 35 kB 00:00 2024-03-30T13:46:49.499 INFO:teuthology.orchestra.run.smithi102.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-30T13:46:50.200 INFO:teuthology.orchestra.run.smithi102.stdout:(22/32): ceph-common-19.0.0-1967.g4e50bbcd.el9. 7.8 MB/s | 21 MB 00:02 2024-03-30T13:46:50.225 INFO:teuthology.orchestra.run.smithi102.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 416 kB/s | 309 kB 00:00 2024-03-30T13:46:50.425 INFO:teuthology.orchestra.run.smithi102.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-03-30T13:46:50.492 INFO:teuthology.orchestra.run.smithi102.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:00 2024-03-30T13:46:50.526 INFO:teuthology.orchestra.run.smithi102.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 162 kB/s | 49 kB 00:00 2024-03-30T13:46:50.584 INFO:teuthology.orchestra.run.smithi102.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-03-30T13:46:50.617 INFO:teuthology.orchestra.run.smithi102.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-30T13:46:50.776 INFO:teuthology.orchestra.run.smithi102.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 764 kB/s | 192 kB 00:00 2024-03-30T13:46:50.944 INFO:teuthology.orchestra.run.smithi102.stdout:(30/32): librados2-19.0.0-1967.g4e50bbcd.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-30T13:46:51.127 INFO:teuthology.orchestra.run.smithi102.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2024-03-30T13:46:51.486 INFO:teuthology.orchestra.run.smithi102.stdout:(32/32): librbd1-19.0.0-1967.g4e50bbcd.el9.x86_ 4.4 MB/s | 3.1 MB 00:00 2024-03-30T13:46:51.490 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:46:51.490 INFO:teuthology.orchestra.run.smithi102.stdout:Total 14 MB/s | 59 MB 00:04 2024-03-30T13:46:51.491 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:46:51.527 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:46:51.527 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:46:52.007 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:46:52.008 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:46:53.509 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:46:53.589 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-30T13:46:53.624 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-30T13:46:53.649 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-30T13:46:53.672 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:46:53.689 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:46:53.881 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:46:53.935 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:46:54.304 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-30T13:46:54.342 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:46:54.433 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:46:54.521 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 8/34 2024-03-30T13:46:54.561 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:46:54.625 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:46:54.651 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-30T13:46:54.694 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 11/34 2024-03-30T13:46:54.763 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:46:54.788 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:46:54.843 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:46:54.878 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-30T13:46:54.985 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-30T13:46:55.029 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-30T13:46:55.070 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-30T13:46:55.102 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-30T13:46:55.131 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-30T13:46:55.163 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-30T13:46:55.475 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-30T13:46:55.502 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:46:55.576 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:46:55.606 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 23/34 2024-03-30T13:46:55.646 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-30T13:46:56.108 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-30T13:46:56.138 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:46:56.200 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:46:56.270 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 27/34 2024-03-30T13:46:56.298 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 28/34 2024-03-30T13:46:58.512 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:46:58.554 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:46:59.097 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:46:59.140 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:46:59.240 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:46:59.241 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T13:46:59.241 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T13:46:59.241 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:46:59.274 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:47:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:47:08.791 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-03-30T13:47:08.792 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-03-30T13:47:08.792 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-03-30T13:47:08.792 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-03-30T13:47:08.792 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-03-30T13:47:08.792 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-03-30T13:47:08.792 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-03-30T13:47:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:47:09.721 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:47:09.722 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:47:09.722 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T13:47:09.722 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:47:09.722 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:47:09.722 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:09.723 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:47:09.809 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:47:09.809 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:47:10.876 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 6/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 8/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 9/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 10/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 11/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 13/34 2024-03-30T13:47:10.877 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-30T13:47:10.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-30T13:47:10.879 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-30T13:47:10.879 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-30T13:47:10.880 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-30T13:47:10.880 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:47:10.880 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-30T13:47:10.880 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/34 2024-03-30T13:47:11.826 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:47:11.826 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:11.826 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-30T13:47:11.826 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.826 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.826 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:47:11.826 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T13:47:11.827 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T13:47:11.828 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T13:47:11.829 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:11.829 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:12.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2024-03-30T13:47:12.641 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:28 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:12.792 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:12.792 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:12.792 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:47:12.792 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 47 M 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 163 k 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-03-30T13:47:12.793 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:12.794 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 48 M 2024-03-30T13:47:12.794 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 205 M 2024-03-30T13:47:12.794 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:47:13.313 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 968 kB/s | 217 kB 00:00 2024-03-30T13:47:13.413 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-30T13:47:13.463 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-30T13:47:13.573 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): libcephsqlite-19.0.0-1967.g4e50bbcd.el9. 338 kB/s | 163 kB 00:00 2024-03-30T13:47:13.598 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-30T13:47:16.723 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): ceph-test-19.0.0-1967.g4e50bbcd.el9.x86_ 13 MB/s | 47 MB 00:03 2024-03-30T13:47:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:47:16.725 INFO:teuthology.orchestra.run.smithi102.stdout:Total 12 MB/s | 48 MB 00:03 2024-03-30T13:47:16.737 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:47:16.755 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:47:16.755 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:47:17.135 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:47:17.135 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:47:17.781 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:47:17.864 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T13:47:17.904 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T13:47:17.944 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:47:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T13:47:18.017 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:47:22.505 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:47:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:47:24.878 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:47:24.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/6 2024-03-30T13:47:24.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/6 2024-03-30T13:47:24.878 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:47:24.879 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T13:47:24.879 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T13:47:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T13:47:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:25.271 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:47:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:25.271 INFO:teuthology.orchestra.run.smithi102.stdout: jq-1.6-16.el9.x86_64 2024-03-30T13:47:25.272 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:25.272 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T13:47:25.272 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T13:47:25.272 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T13:47:25.272 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:25.272 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:25.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2024-03-30T13:47:26.157 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:42 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:26.313 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================= 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================= 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 6.6 k 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 2.4 M 2024-03-30T13:47:26.316 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 1.5 M 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 248 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 4.7 M 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 17 M 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T13:47:26.317 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T13:47:26.318 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================= 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout:Install 36 Packages 2024-03-30T13:47:26.319 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:26.320 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 30 M 2024-03-30T13:47:26.320 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 107 M 2024-03-30T13:47:26.320 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:47:26.919 INFO:teuthology.orchestra.run.smithi102.stdout:(1/36): ceph-19.0.0-1967.g4e50bbcd.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-30T13:47:27.111 INFO:teuthology.orchestra.run.smithi102.stdout:(2/36): ceph-mgr-19.0.0-1967.g4e50bbcd.el9.x86_ 4.1 MB/s | 1.5 MB 00:00 2024-03-30T13:47:27.228 INFO:teuthology.orchestra.run.smithi102.stdout:(3/36): ceph-mds-19.0.0-1967.g4e50bbcd.el9.x86_ 4.9 MB/s | 2.4 MB 00:00 2024-03-30T13:47:27.312 INFO:teuthology.orchestra.run.smithi102.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1967.g4e50 2.9 MB/s | 248 kB 00:00 2024-03-30T13:47:27.429 INFO:teuthology.orchestra.run.smithi102.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-30T13:47:27.470 INFO:teuthology.orchestra.run.smithi102.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-30T13:47:27.504 INFO:teuthology.orchestra.run.smithi102.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 944 kB/s | 32 kB 00:00 2024-03-30T13:47:27.566 INFO:teuthology.orchestra.run.smithi102.stdout:(8/36): ceph-mon-19.0.0-1967.g4e50bbcd.el9.x86_ 7.3 MB/s | 4.7 MB 00:00 2024-03-30T13:47:27.588 INFO:teuthology.orchestra.run.smithi102.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 566 kB/s | 47 kB 00:00 2024-03-30T13:47:27.613 INFO:teuthology.orchestra.run.smithi102.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-30T13:47:27.639 INFO:teuthology.orchestra.run.smithi102.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-30T13:47:27.698 INFO:teuthology.orchestra.run.smithi102.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-30T13:47:27.723 INFO:teuthology.orchestra.run.smithi102.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-30T13:47:27.757 INFO:teuthology.orchestra.run.smithi102.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-30T13:47:27.782 INFO:teuthology.orchestra.run.smithi102.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-30T13:47:27.848 INFO:teuthology.orchestra.run.smithi102.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 334 kB/s | 22 kB 00:00 2024-03-30T13:47:27.907 INFO:teuthology.orchestra.run.smithi102.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 831 kB/s | 125 kB 00:00 2024-03-30T13:47:28.100 INFO:teuthology.orchestra.run.smithi102.stdout:(18/36): ceph-osd-19.0.0-1967.g4e50bbcd.el9.x86 17 MB/s | 17 MB 00:00 2024-03-30T13:47:28.483 INFO:teuthology.orchestra.run.smithi102.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 75 kB/s | 43 kB 00:00 2024-03-30T13:47:28.758 INFO:teuthology.orchestra.run.smithi102.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 166 kB/s | 151 kB 00:00 2024-03-30T13:47:28.867 INFO:teuthology.orchestra.run.smithi102.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-30T13:47:28.942 INFO:teuthology.orchestra.run.smithi102.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 216 kB/s | 172 kB 00:00 2024-03-30T13:47:28.967 INFO:teuthology.orchestra.run.smithi102.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2024-03-30T13:47:29.076 INFO:teuthology.orchestra.run.smithi102.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-30T13:47:29.101 INFO:teuthology.orchestra.run.smithi102.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 580 kB/s | 358 kB 00:00 2024-03-30T13:47:29.118 INFO:teuthology.orchestra.run.smithi102.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 132 kB/s | 23 kB 00:00 2024-03-30T13:47:29.185 INFO:teuthology.orchestra.run.smithi102.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-30T13:47:29.210 INFO:teuthology.orchestra.run.smithi102.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-03-30T13:47:29.394 INFO:teuthology.orchestra.run.smithi102.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 90 kB/s | 16 kB 00:00 2024-03-30T13:47:29.422 INFO:teuthology.orchestra.run.smithi102.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 262 kB/s | 79 kB 00:00 2024-03-30T13:47:29.519 INFO:teuthology.orchestra.run.smithi102.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 815 kB/s | 272 kB 00:00 2024-03-30T13:47:29.636 INFO:teuthology.orchestra.run.smithi102.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 165 kB/s | 36 kB 00:00 2024-03-30T13:47:29.661 INFO:teuthology.orchestra.run.smithi102.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 336 kB/s | 90 kB 00:00 2024-03-30T13:47:29.820 INFO:teuthology.orchestra.run.smithi102.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 764 kB/s | 230 kB 00:00 2024-03-30T13:47:29.862 INFO:teuthology.orchestra.run.smithi102.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 100 kB/s | 20 kB 00:00 2024-03-30T13:47:30.487 INFO:teuthology.orchestra.run.smithi102.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 502 kB/s | 427 kB 00:00 2024-03-30T13:47:30.492 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:47:30.492 INFO:teuthology.orchestra.run.smithi102.stdout:Total 7.2 MB/s | 30 MB 00:04 2024-03-30T13:47:30.517 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:47:30.561 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:47:30.561 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:47:30.969 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:47:30.970 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:47:32.295 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:47:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-30T13:47:32.388 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-30T13:47:32.417 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-30T13:47:32.488 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-30T13:47:32.533 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-30T13:47:32.565 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-30T13:47:32.657 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-30T13:47:32.734 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-30T13:47:32.776 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-30T13:47:32.847 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-30T13:47:32.903 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-30T13:47:32.940 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-30T13:47:33.014 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-30T13:47:33.070 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-30T13:47:33.253 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-30T13:47:33.311 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-30T13:47:33.350 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-30T13:47:33.385 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-30T13:47:33.415 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-30T13:47:33.490 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-30T13:47:33.525 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:47:33.558 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-30T13:47:33.599 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-30T13:47:33.634 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-30T13:47:33.668 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-30T13:47:33.757 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-30T13:47:33.808 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-30T13:47:33.879 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-30T13:47:34.016 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-30T13:47:34.144 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:47:34.319 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 31/36 2024-03-30T13:47:34.346 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:47:34.371 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:47:34.371 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:47:34.371 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T13:47:34.371 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:47:34.371 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:47:34.371 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:35.855 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:47:35.882 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:47:35.882 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:47:35.882 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T13:47:35.882 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:47:35.882 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:47:35.882 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:36.340 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:47:36.363 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:47:36.363 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:47:36.363 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T13:47:36.363 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:47:36.363 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:47:36.363 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:36.607 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:47:36.630 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:47:36.630 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:47:36.630 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T13:47:36.630 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:47:36.630 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:47:36.630 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:36.696 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:47:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 6/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-30T13:47:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-30T13:47:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-30T13:47:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-30T13:47:39.301 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-30T13:47:39.302 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:39.302 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:47:39.302 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:39.302 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:39.302 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:39.302 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:47:39.302 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T13:47:39.303 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T13:47:39.304 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T13:47:39.305 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T13:47:39.306 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T13:47:39.306 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T13:47:39.306 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T13:47:39.306 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:39.306 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:39.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2024-03-30T13:47:40.236 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:56 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:40.333 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:47:40.385 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:40.386 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:47:40.386 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:40.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2024-03-30T13:47:41.066 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:57 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:41.222 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:41.223 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:41.223 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:47:41.223 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:41.223 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:47:41.223 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 748 k 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 748 k 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 755 k 2024-03-30T13:47:41.224 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:47:41.490 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-19.0.0-1967.g4e50bbcd.el9.noarch.rpm 2.8 MB/s | 748 kB 00:00 2024-03-30T13:47:41.491 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:47:41.491 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.7 MB/s | 748 kB 00:00 2024-03-30T13:47:41.491 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:47:41.498 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:47:41.498 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:47:41.502 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:47:41.503 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:47:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:47:41.932 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:47:42.022 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:47:42.402 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:47:42.703 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:47:42.703 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:42.703 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:47:42.703 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:47:42.703 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:42.703 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:42.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T13:47:43.463 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:43.618 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 145 k 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:47:43.619 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-30T13:47:43.620 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:43.620 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 145 k 2024-03-30T13:47:43.620 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 435 k 2024-03-30T13:47:43.620 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:47:43.965 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-19.0.0-1967.g4e50bb 419 kB/s | 145 kB 00:00 2024-03-30T13:47:43.965 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:47:43.965 INFO:teuthology.orchestra.run.smithi102.stdout:Total 418 kB/s | 145 kB 00:00 2024-03-30T13:47:43.966 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:47:43.973 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:47:43.973 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:47:44.025 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:47:44.026 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:47:44.166 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:47:44.216 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:47:44.240 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:47:44.240 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:47:44.240 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T13:47:44.240 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:47:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:45.214 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:47:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:47:45.214 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:45.214 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:45.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2024-03-30T13:47:46.049 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:46.146 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:47:46.198 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:46.199 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:47:46.199 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:46.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T13:47:46.839 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:46.986 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:46.987 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-03-30T13:47:46.987 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:47:46.987 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-03-30T13:47:46.987 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:47:46.987 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 3.5 M 2024-03-30T13:47:46.987 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:47:46.987 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 31 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 15 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:47:46.988 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-03-30T13:47:46.989 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-03-30T13:47:46.989 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:46.989 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 8.6 M 2024-03-30T13:47:46.989 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 103 M 2024-03-30T13:47:46.989 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:47:47.469 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1967.g4e 165 kB/s | 31 kB 00:00 2024-03-30T13:47:47.577 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1967.g4e5 51 kB/s | 15 kB 00:00 2024-03-30T13:47:47.636 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-03-30T13:47:47.661 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-30T13:47:47.986 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1967.g4e50bbc 5.0 MB/s | 3.5 MB 00:00 2024-03-30T13:47:48.042 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 71 kB/s | 6.1 MB 01:28 2024-03-30T13:47:48.045 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.1 MB/s | 863 kB 00:00 2024-03-30T13:47:48.237 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-30T13:47:48.296 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 864 kB/s | 548 kB 00:00 2024-03-30T13:47:48.396 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-30T13:47:48.521 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-30T13:47:48.588 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 412 kB/s | 144 kB 00:00 2024-03-30T13:47:48.830 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.6 MB/s | 2.0 MB 00:00 2024-03-30T13:47:48.833 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:47:48.833 INFO:teuthology.orchestra.run.smithi102.stdout:Total 4.7 MB/s | 8.6 MB 00:01 2024-03-30T13:47:48.853 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:47:48.872 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:47:48.872 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:47:49.056 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:47:49.057 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:47:49.593 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:47:49.664 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-30T13:47:49.699 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-30T13:47:49.701 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-30T13:47:49.746 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-30T13:47:49.793 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-30T13:47:49.934 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-30T13:47:50.102 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-30T13:47:50.204 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-30T13:47:50.255 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-30T13:47:50.284 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:47:50.316 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 10/12 2024-03-30T13:47:51.361 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 11/12 2024-03-30T13:47:51.400 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:47:52.719 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:47:52.719 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 1/12 2024-03-30T13:47:52.719 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 2/12 2024-03-30T13:47:52.719 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 3/12 2024-03-30T13:47:52.719 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-30T13:47:52.719 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-30T13:47:52.719 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-30T13:47:52.720 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-30T13:47:52.720 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-30T13:47:52.720 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:47:52.720 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-30T13:47:52.720 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-30T13:47:52.945 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-03-30T13:47:53.059 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-30T13:47:53.059 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:53.059 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:47:53.059 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-30T13:47:53.059 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:53.060 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:47:53.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T13:47:53.878 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:09 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:47:54.030 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:47:54.031 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-30T13:47:54.031 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:47:54.031 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-30T13:47:54.031 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:47:54.031 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 7.4 M 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:47:54.032 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-30T13:47:54.033 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-03-30T13:47:54.033 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:47:54.033 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 37 M 2024-03-30T13:47:54.033 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 209 M 2024-03-30T13:47:54.033 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:47:54.388 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 179 kB/s | 32 kB 00:00 2024-03-30T13:47:54.447 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 295 kB/s | 17 kB 00:00 2024-03-30T13:47:54.538 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 372 kB/s | 34 kB 00:00 2024-03-30T13:47:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): ceph-mgr-diskprediction-local-19.0.0-19 11 MB/s | 7.4 MB 00:00 2024-03-30T13:47:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-03-30T13:47:55.298 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 607 kB/s | 217 kB 00:00 2024-03-30T13:47:55.381 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.6 MB/s | 4.7 MB 00:00 2024-03-30T13:47:55.507 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.4 MB/s | 803 kB 00:00 2024-03-30T13:47:55.599 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-30T13:47:55.658 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.3 MB/s | 4.8 MB 00:00 2024-03-30T13:47:56.232 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-03-30T13:47:56.683 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 227 kB/s | 245 kB 00:01 2024-03-30T13:47:56.686 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:47:56.686 INFO:teuthology.orchestra.run.smithi102.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-30T13:47:56.728 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:47:56.753 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:47:56.753 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:47:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:47:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:47:57.813 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:47:57.894 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-30T13:47:57.897 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T13:47:57.937 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T13:47:58.030 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:47:58.065 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T13:47:58.242 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:47:58.281 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:47:58.357 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T13:47:59.017 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T13:47:59.118 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T13:48:00.360 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T13:48:00.478 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 9.8 M 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.4 M 2024-03-30T13:48:00.494 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.3 M 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 21 M 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 25 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 745 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T13:48:00.495 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 501 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.1 M 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 45 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 131 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 161 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 321 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 298 k 2024-03-30T13:48:00.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 99 k 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout:Installing weak dependencies: 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout:Install 30 Packages 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 2 Packages 2024-03-30T13:48:00.497 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:00.498 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 59 M 2024-03-30T13:48:00.498 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:48:01.312 INFO:teuthology.orchestra.run.smithi138.stdout:(1/32): ceph-base-19.0.0-1967.g4e50bbcd.el9.x86 8.7 MB/s | 5.3 MB 00:00 2024-03-30T13:48:01.362 INFO:teuthology.orchestra.run.smithi138.stdout:(2/32): ceph-selinux-19.0.0-1967.g4e50bbcd.el9. 505 kB/s | 25 kB 00:00 2024-03-30T13:48:01.381 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T13:48:01.414 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:48:01.470 INFO:teuthology.orchestra.run.smithi138.stdout:(3/32): libcephfs2-19.0.0-1967.g4e50bbcd.el9.x8 6.7 MB/s | 745 kB 00:00 2024-03-30T13:48:01.554 INFO:teuthology.orchestra.run.smithi138.stdout:(4/32): libradosstriper1-19.0.0-1967.g4e50bbcd. 5.9 MB/s | 501 kB 00:00 2024-03-30T13:48:02.029 INFO:teuthology.orchestra.run.smithi138.stdout:(5/32): librgw2-19.0.0-1967.g4e50bbcd.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-30T13:48:02.096 INFO:teuthology.orchestra.run.smithi138.stdout:(6/32): python3-ceph-argparse-19.0.0-1967.g4e50 679 kB/s | 45 kB 00:00 2024-03-30T13:48:02.171 INFO:teuthology.orchestra.run.smithi138.stdout:(7/32): python3-ceph-common-19.0.0-1967.g4e50bb 1.7 MB/s | 131 kB 00:00 2024-03-30T13:48:02.289 INFO:teuthology.orchestra.run.smithi138.stdout:(8/32): ceph-radosgw-19.0.0-1967.g4e50bbcd.el9. 6.1 MB/s | 9.8 MB 00:01 2024-03-30T13:48:02.314 INFO:teuthology.orchestra.run.smithi138.stdout:(9/32): python3-cephfs-19.0.0-1967.g4e50bbcd.el 1.1 MB/s | 161 kB 00:00 2024-03-30T13:48:02.372 INFO:teuthology.orchestra.run.smithi138.stdout:(10/32): python3-rbd-19.0.0-1967.g4e50bbcd.el9. 5.0 MB/s | 298 kB 00:00 2024-03-30T13:48:02.406 INFO:teuthology.orchestra.run.smithi138.stdout:(11/32): python3-rados-19.0.0-1967.g4e50bbcd.el 2.7 MB/s | 321 kB 00:00 2024-03-30T13:48:02.431 INFO:teuthology.orchestra.run.smithi138.stdout:(12/32): python3-rgw-19.0.0-1967.g4e50bbcd.el9. 1.7 MB/s | 99 kB 00:00 2024-03-30T13:48:02.573 INFO:teuthology.orchestra.run.smithi138.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-30T13:48:02.598 INFO:teuthology.orchestra.run.smithi138.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2024-03-30T13:48:02.623 INFO:teuthology.orchestra.run.smithi138.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-30T13:48:02.874 INFO:teuthology.orchestra.run.smithi138.stdout:(16/32): ceph-common-19.0.0-1967.g4e50bbcd.el9. 9.4 MB/s | 21 MB 00:02 2024-03-30T13:48:02.907 INFO:teuthology.orchestra.run.smithi138.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 661 kB 00:00 2024-03-30T13:48:02.932 INFO:teuthology.orchestra.run.smithi138.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 915 kB/s | 282 kB 00:00 2024-03-30T13:48:02.958 INFO:teuthology.orchestra.run.smithi138.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 500 kB/s | 41 kB 00:00 2024-03-30T13:48:02.975 INFO:teuthology.orchestra.run.smithi138.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 834 kB/s | 35 kB 00:00 2024-03-30T13:48:03.000 INFO:teuthology.orchestra.run.smithi138.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-30T13:48:03.017 INFO:teuthology.orchestra.run.smithi138.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 787 kB/s | 46 kB 00:00 2024-03-30T13:48:03.459 INFO:teuthology.orchestra.run.smithi138.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 59 kB/s | 26 kB 00:00 2024-03-30T13:48:03.684 INFO:teuthology.orchestra.run.smithi138.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 215 kB/s | 49 kB 00:00 2024-03-30T13:48:03.868 INFO:teuthology.orchestra.run.smithi138.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 346 kB/s | 309 kB 00:00 2024-03-30T13:48:03.918 INFO:teuthology.orchestra.run.smithi138.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 288 kB/s | 67 kB 00:00 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 1/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T13:48:03.951 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T13:48:03.952 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T13:48:04.185 INFO:teuthology.orchestra.run.smithi138.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T13:48:04.410 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:04.411 INFO:teuthology.orchestra.run.smithi138.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 389 kB/s | 192 kB 00:00 2024-03-30T13:48:04.452 INFO:teuthology.orchestra.run.smithi138.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-03-30T13:48:04.719 INFO:teuthology.orchestra.run.smithi138.stdout:(30/32): librados2-19.0.0-1967.g4e50bbcd.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-30T13:48:04.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2024-03-30T13:48:04.902 INFO:teuthology.orchestra.run.smithi138.stdout:(31/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 810 kB/s | 838 kB 00:01 2024-03-30T13:48:05.095 INFO:teuthology.orchestra.run.smithi138.stdout:(32/32): librbd1-19.0.0-1967.g4e50bbcd.el9.x86_ 4.9 MB/s | 3.1 MB 00:00 2024-03-30T13:48:05.102 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:05.102 INFO:teuthology.orchestra.run.smithi138.stdout:Total 13 MB/s | 59 MB 00:04 2024-03-30T13:48:05.103 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:48:05.142 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:48:05.142 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:48:05.425 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:21 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:05.578 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:05.579 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 49 k 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T13:48:05.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2024-03-30T13:48:05.581 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:05.582 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.0 M 2024-03-30T13:48:05.582 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 26 M 2024-03-30T13:48:05.582 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:05.622 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:48:05.622 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:48:05.931 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 150 kB/s | 24 kB 00:00 2024-03-30T13:48:05.947 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 100 kB/s | 18 kB 00:00 2024-03-30T13:48:06.081 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-03-30T13:48:06.106 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 837 kB/s | 132 kB 00:00 2024-03-30T13:48:06.157 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-30T13:48:06.182 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 567 kB/s | 43 kB 00:00 2024-03-30T13:48:06.207 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): ceph-mgr-rook-19.0.0-1967.g4e50bbcd.el9 112 kB/s | 49 kB 00:00 2024-03-30T13:48:06.399 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-03-30T13:48:06.425 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-03-30T13:48:06.533 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-30T13:48:06.584 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 539 kB/s | 203 kB 00:00 2024-03-30T13:48:06.634 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 895 kB/s | 90 kB 00:00 2024-03-30T13:48:06.734 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-30T13:48:06.738 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:06.738 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-30T13:48:06.738 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:06.751 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:06.751 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:06.841 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:06.842 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:07.105 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:48:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-30T13:48:07.224 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-30T13:48:07.249 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-30T13:48:07.272 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:48:07.289 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T13:48:07.351 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:07.482 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:48:07.515 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T13:48:07.537 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T13:48:07.568 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T13:48:07.632 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T13:48:07.669 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T13:48:07.711 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T13:48:07.804 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T13:48:07.882 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-30T13:48:07.912 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T13:48:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T13:48:07.976 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:48:08.061 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:48:08.149 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 8/34 2024-03-30T13:48:08.186 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:48:08.249 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T13:48:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-30T13:48:08.321 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 11/34 2024-03-30T13:48:08.391 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:48:08.413 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:48:08.460 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T13:48:08.495 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-30T13:48:08.607 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-30T13:48:08.647 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-30T13:48:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T13:48:08.686 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-30T13:48:08.719 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-30T13:48:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T13:48:08.748 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-30T13:48:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T13:48:08.781 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-30T13:48:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T13:48:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:48:09.099 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-30T13:48:09.136 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:48:09.225 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T13:48:09.249 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 23/34 2024-03-30T13:48:09.289 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-30T13:48:09.696 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:48:09.696 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/13 2024-03-30T13:48:09.696 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T13:48:09.696 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T13:48:09.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T13:48:09.743 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-30T13:48:09.772 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:48:09.826 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T13:48:09.894 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 27/34 2024-03-30T13:48:09.924 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 28/34 2024-03-30T13:48:10.071 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T13:48:10.072 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T13:48:10.073 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T13:48:10.073 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T13:48:10.073 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:10.073 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:10.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T13:48:10.882 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:11.037 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:11.038 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-30T13:48:11.038 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:48:11.038 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-30T13:48:11.038 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:11.038 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 139 k 2024-03-30T13:48:11.038 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:48:11.038 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-03-30T13:48:11.039 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:11.040 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 6.8 M 2024-03-30T13:48:11.040 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 33 M 2024-03-30T13:48:11.040 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:11.604 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1967.g4e50bbcd.e 563 kB/s | 139 kB 00:00 2024-03-30T13:48:11.770 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-30T13:48:11.871 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2024-03-30T13:48:11.913 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-30T13:48:11.954 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 316 kB/s | 58 kB 00:00 2024-03-30T13:48:12.191 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:48:12.229 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:48:12.688 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 186 kB/s | 248 kB 00:01 2024-03-30T13:48:12.690 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:12.690 INFO:teuthology.orchestra.run.smithi102.stdout:Total 4.1 MB/s | 6.8 MB 00:01 2024-03-30T13:48:12.705 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:12.717 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:12.717 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:12.777 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T13:48:12.809 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:12.811 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:12.815 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:48:12.905 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T13:48:12.906 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T13:48:12.906 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T13:48:12.906 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:12.958 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:48:13.107 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:13.295 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T13:48:13.340 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T13:48:13.849 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T13:48:13.934 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T13:48:13.996 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T13:48:14.019 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:48:14.778 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:48:14.779 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/6 2024-03-30T13:48:14.779 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T13:48:14.779 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T13:48:14.779 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T13:48:14.779 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T13:48:15.092 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T13:48:15.092 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:15.092 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:15.093 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:15.360 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2024-03-30T13:48:15.918 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:16.070 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:16.070 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:16.070 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 889 k 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-03-30T13:48:16.071 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:16.072 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 968 k 2024-03-30T13:48:16.072 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.8 M 2024-03-30T13:48:16.072 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:16.214 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 552 kB/s | 79 kB 00:00 2024-03-30T13:48:16.448 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-fuse-19.0.0-1967.g4e50bbcd.el9.x86_ 2.3 MB/s | 889 kB 00:00 2024-03-30T13:48:16.449 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:16.450 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.5 MB/s | 968 kB 00:00 2024-03-30T13:48:16.450 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:16.471 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:16.471 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:16.543 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:16.544 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:16.867 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T13:48:16.932 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:48:17.564 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:48:17.564 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/2 2024-03-30T13:48:17.860 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T13:48:17.860 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:17.860 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:17.861 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T13:48:17.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:17.861 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:18.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-volume 2024-03-30T13:48:18.718 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:18.874 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:18.874 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 264 k 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:18.875 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 332 k 2024-03-30T13:48:18.876 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 1.5 M 2024-03-30T13:48:18.876 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:19.003 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 539 kB/s | 69 kB 00:00 2024-03-30T13:48:19.262 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-volume-19.0.0-1967.g4e50bbcd.el9.no 683 kB/s | 264 kB 00:00 2024-03-30T13:48:19.263 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:19.263 INFO:teuthology.orchestra.run.smithi102.stdout:Total 856 kB/s | 332 kB 00:00 2024-03-30T13:48:19.263 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:19.272 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:19.272 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:19.312 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:19.312 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:19.464 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:19.651 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T13:48:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:48:19.716 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:48:19.716 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:19.716 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T13:48:19.717 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:20.481 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/2 2024-03-30T13:48:20.819 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T13:48:20.819 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:20.819 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:20.819 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:48:20.819 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T13:48:20.819 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:20.819 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:21.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2024-03-30T13:48:21.728 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:37 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:21.883 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:21.883 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 127 k 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 127 k 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 456 k 2024-03-30T13:48:21.884 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:22.137 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-19.0.0-1967.g4e50bbcd.el9.x86_64 504 kB/s | 127 kB 00:00 2024-03-30T13:48:22.137 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:22.138 INFO:teuthology.orchestra.run.smithi102.stdout:Total 501 kB/s | 127 kB 00:00 2024-03-30T13:48:22.138 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:22.143 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:22.143 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:22.198 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:22.199 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:22.344 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:22.460 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:48:22.817 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2024-03-30T13:48:22.817 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2024-03-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2024-03-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2024-03-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2024-03-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /root 2024-03-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2024-03-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:22.922 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:23.207 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:23.208 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:23.208 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:23.208 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:23.208 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:23.208 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:23.444 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2024-03-30T13:48:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:48:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T13:48:23.725 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:23.726 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T13:48:23.726 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:48:23.726 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T13:48:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:48:23.821 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T13:48:23.821 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:48:24.015 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:24.115 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:24.170 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:24.171 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:48:24.172 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:24.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2024-03-30T13:48:24.802 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:24.954 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:24.955 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:24.955 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-30T13:48:24.955 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 32 k 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 32 k 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 142 k 2024-03-30T13:48:24.956 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 6/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 8/34 2024-03-30T13:48:24.980 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 9/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 10/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 11/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 13/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-30T13:48:24.981 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-30T13:48:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/34 2024-03-30T13:48:25.440 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-19.0.0-1967.g4e50bbcd.el9.x86_6 65 kB/s | 32 kB 00:00 2024-03-30T13:48:25.441 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:25.441 INFO:teuthology.orchestra.run.smithi102.stdout:Total 65 kB/s | 32 kB 00:00 2024-03-30T13:48:25.441 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:25.445 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:25.445 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:25.477 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:25.477 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:25.606 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:25.689 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:26.065 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:26.159 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T13:48:26.159 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.160 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T13:48:26.161 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:26.162 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:48:26.338 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:26.338 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:26.339 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:26.339 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:26.339 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:26.339 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:26.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2024-03-30T13:48:26.514 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-test 2024-03-30T13:48:27.069 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:27.071 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:30 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:48:27.168 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:27.219 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:27.220 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:48:27.220 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:27.220 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:48:27.221 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:27.221 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:48:27.221 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:27.221 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 47 M 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 163 k 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-03-30T13:48:27.222 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:27.223 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 48 M 2024-03-30T13:48:27.223 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 205 M 2024-03-30T13:48:27.223 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:48:27.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2024-03-30T13:48:27.440 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-30T13:48:27.499 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-30T13:48:27.532 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): libcephsqlite-19.0.0-1967.g4e50bbcd.el9. 596 kB/s | 163 kB 00:00 2024-03-30T13:48:27.566 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 932 kB/s | 63 kB 00:00 2024-03-30T13:48:27.758 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 831 kB/s | 187 kB 00:00 2024-03-30T13:48:27.841 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:27.938 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:27.989 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:27.990 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:48:27.990 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:28.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2024-03-30T13:48:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:28.700 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:28.752 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:28.752 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:48:28.753 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:28.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2024-03-30T13:48:29.377 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:29.475 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:29.529 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:29.530 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:48:29.530 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:29.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2024-03-30T13:48:29.792 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): ceph-test-19.0.0-1967.g4e50bbcd.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-30T13:48:29.795 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:29.795 INFO:teuthology.orchestra.run.smithi138.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-30T13:48:29.810 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:48:29.832 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:48:29.832 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:48:30.163 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:30.214 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:48:30.215 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:48:30.261 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:30.313 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:30.314 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:48:30.314 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:30.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2024-03-30T13:48:30.804 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:48:30.902 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T13:48:30.930 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:30.943 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T13:48:30.977 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:48:31.018 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T13:48:31.028 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:31.045 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:48:31.079 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:31.080 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-30T13:48:31.080 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:31.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2024-03-30T13:48:31.706 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:47 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:31.857 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 85 k 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-30T13:48:31.858 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:31.859 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 85 k 2024-03-30T13:48:31.859 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 231 k 2024-03-30T13:48:31.859 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:32.068 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 408 kB/s | 85 kB 00:00 2024-03-30T13:48:32.069 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:32.069 INFO:teuthology.orchestra.run.smithi102.stdout:Total 405 kB/s | 85 kB 00:00 2024-03-30T13:48:32.069 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:32.076 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:32.076 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:32.133 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:32.133 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:32.266 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:32.746 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:33.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:33.038 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:33.038 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:33.038 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:33.038 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:33.038 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:33.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2024-03-30T13:48:33.854 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:49 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:34.009 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:34.010 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:34.010 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-30T13:48:34.010 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:34.010 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.1 M 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 13 M 2024-03-30T13:48:34.011 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:34.422 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 7.5 MB/s | 3.1 MB 00:00 2024-03-30T13:48:34.422 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:34.422 INFO:teuthology.orchestra.run.smithi102.stdout:Total 7.5 MB/s | 3.1 MB 00:00 2024-03-30T13:48:34.423 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:34.431 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:34.432 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:34.505 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:34.505 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:34.931 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:34.981 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:35.005 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:35.006 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:35.006 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T13:48:35.006 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:48:35.006 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:48:35.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:35.538 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T13:48:35.613 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:48:36.115 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:36.115 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:36.115 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:36.115 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:36.115 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:36.115 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:36.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2024-03-30T13:48:36.950 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:52 ago on Sat 30 Mar 2024 01:46:44 PM UTC. 2024-03-30T13:48:37.102 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 170 k 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 170 k 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 494 k 2024-03-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-30T13:48:37.326 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 772 kB/s | 170 kB 00:00 2024-03-30T13:48:37.326 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:37.326 INFO:teuthology.orchestra.run.smithi102.stdout:Total 768 kB/s | 170 kB 00:00 2024-03-30T13:48:37.326 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-30T13:48:37.333 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-30T13:48:37.334 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-30T13:48:37.389 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-30T13:48:37.389 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-30T13:48:37.541 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-30T13:48:37.656 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:38.069 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:38.071 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T13:48:38.071 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/6 2024-03-30T13:48:38.071 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/6 2024-03-30T13:48:38.072 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T13:48:38.072 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T13:48:38.072 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T13:48:38.355 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:48:38.355 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:38.355 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-30T13:48:38.355 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:38.355 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:48:38.355 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-30T13:48:38.454 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T13:48:38.454 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout: jq-1.6-16.el9.x86_64 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:38.455 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:48:38.604 DEBUG:teuthology.parallel:result is None 2024-03-30T13:48:38.789 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph 2024-03-30T13:48:39.360 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:42 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:48:39.522 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 6.6 k 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 2.4 M 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 1.5 M 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 248 k 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 4.7 M 2024-03-30T13:48:39.525 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 17 M 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T13:48:39.526 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T13:48:39.527 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout:========================================================================================= 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout:Install 36 Packages 2024-03-30T13:48:39.528 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:39.529 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 30 M 2024-03-30T13:48:39.529 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 107 M 2024-03-30T13:48:39.529 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:48:40.059 INFO:teuthology.orchestra.run.smithi138.stdout:(1/36): ceph-19.0.0-1967.g4e50bbcd.el9.x86_64.r 26 kB/s | 6.6 kB 00:00 2024-03-30T13:48:41.117 INFO:teuthology.orchestra.run.smithi138.stdout:(2/36): ceph-mgr-19.0.0-1967.g4e50bbcd.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-03-30T13:48:41.293 INFO:teuthology.orchestra.run.smithi138.stdout:(3/36): ceph-mds-19.0.0-1967.g4e50bbcd.el9.x86_ 1.6 MB/s | 2.4 MB 00:01 2024-03-30T13:48:41.360 INFO:teuthology.orchestra.run.smithi138.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1967.g4e50 3.7 MB/s | 248 kB 00:00 2024-03-30T13:48:41.477 INFO:teuthology.orchestra.run.smithi138.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-30T13:48:41.519 INFO:teuthology.orchestra.run.smithi138.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-30T13:48:41.561 INFO:teuthology.orchestra.run.smithi138.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 754 kB/s | 32 kB 00:00 2024-03-30T13:48:41.603 INFO:teuthology.orchestra.run.smithi138.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-30T13:48:41.637 INFO:teuthology.orchestra.run.smithi138.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-30T13:48:41.687 INFO:teuthology.orchestra.run.smithi138.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-30T13:48:41.773 INFO:teuthology.orchestra.run.smithi138.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-30T13:48:41.812 INFO:teuthology.orchestra.run.smithi138.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-30T13:48:41.855 INFO:teuthology.orchestra.run.smithi138.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-30T13:48:41.897 INFO:teuthology.orchestra.run.smithi138.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-30T13:48:41.964 INFO:teuthology.orchestra.run.smithi138.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-03-30T13:48:42.064 INFO:teuthology.orchestra.run.smithi138.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-30T13:48:42.181 INFO:teuthology.orchestra.run.smithi138.stdout:(17/36): ceph-mon-19.0.0-1967.g4e50bbcd.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-03-30T13:48:42.523 INFO:teuthology.orchestra.run.smithi138.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-03-30T13:48:42.549 INFO:teuthology.orchestra.run.smithi138.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 312 kB/s | 151 kB 00:00 2024-03-30T13:48:42.774 INFO:teuthology.orchestra.run.smithi138.stdout:(20/36): ceph-osd-19.0.0-1967.g4e50bbcd.el9.x86 10 MB/s | 17 MB 00:01 2024-03-30T13:48:42.958 INFO:teuthology.orchestra.run.smithi138.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-03-30T13:48:43.117 INFO:teuthology.orchestra.run.smithi138.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 290 kB/s | 172 kB 00:00 2024-03-30T13:48:43.217 INFO:teuthology.orchestra.run.smithi138.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 68 kB/s | 18 kB 00:00 2024-03-30T13:48:43.275 INFO:teuthology.orchestra.run.smithi138.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 493 kB/s | 358 kB 00:00 2024-03-30T13:48:43.376 INFO:teuthology.orchestra.run.smithi138.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 89 kB/s | 23 kB 00:00 2024-03-30T13:48:43.476 INFO:teuthology.orchestra.run.smithi138.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 75 kB/s | 19 kB 00:00 2024-03-30T13:48:43.552 INFO:teuthology.orchestra.run.smithi138.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 71 kB/s | 20 kB 00:00 2024-03-30T13:48:43.735 INFO:teuthology.orchestra.run.smithi138.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 129 kB/s | 46 kB 00:00 2024-03-30T13:48:43.910 INFO:teuthology.orchestra.run.smithi138.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 182 kB/s | 79 kB 00:00 2024-03-30T13:48:44.002 INFO:teuthology.orchestra.run.smithi138.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-03-30T13:48:44.053 INFO:teuthology.orchestra.run.smithi138.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 544 kB/s | 272 kB 00:00 2024-03-30T13:48:44.270 INFO:teuthology.orchestra.run.smithi138.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 250 kB/s | 90 kB 00:00 2024-03-30T13:48:44.287 INFO:teuthology.orchestra.run.smithi138.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 126 kB/s | 36 kB 00:00 2024-03-30T13:48:44.554 INFO:teuthology.orchestra.run.smithi138.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-03-30T13:48:44.579 INFO:teuthology.orchestra.run.smithi138.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 436 kB/s | 230 kB 00:00 2024-03-30T13:48:44.854 INFO:teuthology.orchestra.run.smithi138.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 731 kB/s | 427 kB 00:00 2024-03-30T13:48:44.859 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:44.859 INFO:teuthology.orchestra.run.smithi138.stdout:Total 5.6 MB/s | 30 MB 00:05 2024-03-30T13:48:44.885 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:48:44.926 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:48:44.927 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:48:45.330 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:48:45.331 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:48:46.657 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:48:46.733 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-30T13:48:46.765 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-30T13:48:46.793 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-30T13:48:46.864 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-30T13:48:46.901 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-30T13:48:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-30T13:48:47.025 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-30T13:48:47.095 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-30T13:48:47.137 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-30T13:48:47.209 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-30T13:48:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-30T13:48:47.302 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-30T13:48:47.351 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-30T13:48:47.406 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-30T13:48:47.592 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-30T13:48:47.639 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-30T13:48:47.670 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-30T13:48:47.705 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-30T13:48:47.735 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-30T13:48:47.812 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-30T13:48:47.846 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:48:47.879 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-30T13:48:47.920 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-30T13:48:47.955 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-30T13:48:47.998 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-30T13:48:48.087 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-30T13:48:48.140 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-30T13:48:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-30T13:48:48.355 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-30T13:48:48.474 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:48:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 31/36 2024-03-30T13:48:48.660 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:48:48.686 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T13:48:48.686 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:48.686 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T13:48:48.686 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:48:48.686 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T13:48:48.686 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:50.128 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:48:50.154 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T13:48:50.154 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:50.155 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T13:48:50.155 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:48:50.155 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T13:48:50.155 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:48:50.619 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T13:48:50.619 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:50.619 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T13:48:50.619 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:48:50.620 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T13:48:50.620 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:50.847 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:48:50.870 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T13:48:50.870 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:50.870 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T13:48:50.870 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:48:50.870 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T13:48:50.871 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:50.936 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:48:52.812 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T13:48:52.812 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/36 2024-03-30T13:48:52.812 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 6/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-30T13:48:52.813 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-30T13:48:52.814 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-30T13:48:52.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-30T13:48:52.815 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-30T13:48:52.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T13:48:52.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-30T13:48:52.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-30T13:48:52.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-30T13:48:52.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-30T13:48:52.816 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-30T13:48:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-30T13:48:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:53.871 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:48:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:53.871 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: lua-5.4.4-4.el9.x86_64 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T13:48:53.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:48:54.238 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-base 2024-03-30T13:48:54.824 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:57 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:48:54.924 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:48:54.977 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:48:54.978 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:48:54.978 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:48:55.042 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install cephadm 2024-03-30T13:48:55.597 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:58 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:48:55.746 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 748 k 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 748 k 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 755 k 2024-03-30T13:48:55.747 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:48:56.425 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm-19.0.0-1967.g4e50bbcd.el9.noarch.rpm 1.1 MB/s | 748 kB 00:00 2024-03-30T13:48:56.425 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:56.425 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.1 MB/s | 748 kB 00:00 2024-03-30T13:48:56.425 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:48:56.430 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:48:56.431 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:48:56.436 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:48:56.437 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:48:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:48:56.934 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:48:57.034 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:48:57.403 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:48:57.693 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T13:48:57.693 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:57.693 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:48:57.693 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:48:57.693 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:57.693 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:48:57.906 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T13:48:58.482 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:01 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:48:58.638 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 145 k 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:48:58.639 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 145 k 2024-03-30T13:48:58.640 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 435 k 2024-03-30T13:48:58.640 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:48:58.956 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-immutable-object-cache-19.0.0-1967.g4e50bb 458 kB/s | 145 kB 00:00 2024-03-30T13:48:58.956 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:48:58.956 INFO:teuthology.orchestra.run.smithi138.stdout:Total 456 kB/s | 145 kB 00:00 2024-03-30T13:48:58.956 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:48:58.964 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:48:58.964 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:48:59.016 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:48:59.016 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:48:59.161 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:48:59.220 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:48:59.245 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:48:59.245 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:48:59.245 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T13:48:59.245 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:00.242 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T13:49:00.242 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:00.242 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:00.243 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:49:00.243 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:00.243 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:00.507 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr 2024-03-30T13:49:01.081 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:04 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:01.180 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:01.233 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:01.234 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:01.234 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:01.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T13:49:01.869 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:04 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:02.023 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:02.024 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-30T13:49:02.024 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:49:02.024 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-30T13:49:02.024 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:02.024 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 3.5 M 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 31 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 15 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:02.025 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:02.026 INFO:teuthology.orchestra.run.smithi138.stdout:====================================================================================== 2024-03-30T13:49:02.026 INFO:teuthology.orchestra.run.smithi138.stdout:Install 12 Packages 2024-03-30T13:49:02.026 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:02.026 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 8.6 M 2024-03-30T13:49:02.026 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 103 M 2024-03-30T13:49:02.026 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:02.701 INFO:teuthology.orchestra.run.smithi138.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1967.g4e 129 kB/s | 31 kB 00:00 2024-03-30T13:49:02.726 INFO:teuthology.orchestra.run.smithi138.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1967.g4e5 57 kB/s | 15 kB 00:00 2024-03-30T13:49:02.868 INFO:teuthology.orchestra.run.smithi138.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-30T13:49:02.901 INFO:teuthology.orchestra.run.smithi138.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-30T13:49:03.085 INFO:teuthology.orchestra.run.smithi138.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-03-30T13:49:03.144 INFO:teuthology.orchestra.run.smithi138.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1967.g4e50bbc 5.1 MB/s | 3.5 MB 00:00 2024-03-30T13:49:03.461 INFO:teuthology.orchestra.run.smithi138.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 52 kB/s | 19 kB 00:00 2024-03-30T13:49:03.694 INFO:teuthology.orchestra.run.smithi138.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 691 kB/s | 548 kB 00:00 2024-03-30T13:49:03.778 INFO:teuthology.orchestra.run.smithi138.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 456 kB/s | 144 kB 00:00 2024-03-30T13:49:03.804 INFO:teuthology.orchestra.run.smithi138.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-30T13:49:03.979 INFO:teuthology.orchestra.run.smithi138.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 941 kB/s | 188 kB 00:00 2024-03-30T13:49:04.071 INFO:teuthology.orchestra.run.smithi138.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-30T13:49:04.077 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:04.077 INFO:teuthology.orchestra.run.smithi138.stdout:Total 4.2 MB/s | 8.6 MB 00:02 2024-03-30T13:49:04.096 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:04.113 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:04.113 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:04.300 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:04.300 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:04.829 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-30T13:49:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-30T13:49:04.987 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-30T13:49:05.018 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-30T13:49:05.126 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-30T13:49:05.251 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-30T13:49:05.345 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-30T13:49:05.407 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-30T13:49:05.436 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:49:05.467 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 10/12 2024-03-30T13:49:06.501 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 11/12 2024-03-30T13:49:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 1/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 2/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 3/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-30T13:49:08.051 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-30T13:49:08.052 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-30T13:49:08.052 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-30T13:49:08.393 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:08.394 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:08.674 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T13:49:09.237 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:12 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:09.389 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:09.390 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 7.4 M 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T13:49:09.391 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout:Install 12 Packages 2024-03-30T13:49:09.392 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:09.393 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 37 M 2024-03-30T13:49:09.393 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 209 M 2024-03-30T13:49:09.393 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:10.091 INFO:teuthology.orchestra.run.smithi138.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 220 kB/s | 32 kB 00:00 2024-03-30T13:49:10.124 INFO:teuthology.orchestra.run.smithi138.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 532 kB/s | 17 kB 00:00 2024-03-30T13:49:10.166 INFO:teuthology.orchestra.run.smithi138.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 830 kB/s | 34 kB 00:00 2024-03-30T13:49:10.225 INFO:teuthology.orchestra.run.smithi138.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-30T13:49:10.451 INFO:teuthology.orchestra.run.smithi138.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-03-30T13:49:10.526 INFO:teuthology.orchestra.run.smithi138.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-30T13:49:10.551 INFO:teuthology.orchestra.run.smithi138.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-30T13:49:10.951 INFO:teuthology.orchestra.run.smithi138.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-30T13:49:11.752 INFO:teuthology.orchestra.run.smithi138.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 4.1 MB/s | 7.4 MB 00:01 2024-03-30T13:49:11.919 INFO:teuthology.orchestra.run.smithi138.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-30T13:49:12.744 INFO:teuthology.orchestra.run.smithi138.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 109 kB/s | 195 kB 00:01 2024-03-30T13:49:21.851 INFO:teuthology.orchestra.run.smithi138.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 71 kB/s | 803 kB 00:11 2024-03-30T13:49:21.853 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:21.854 INFO:teuthology.orchestra.run.smithi138.stdout:Total 3.0 MB/s | 37 MB 00:12 2024-03-30T13:49:21.898 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:21.923 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:21.923 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:22.299 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:22.299 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:22.997 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T13:49:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T13:49:23.325 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:49:23.359 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T13:49:23.538 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:49:23.567 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:49:23.644 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T13:49:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T13:49:24.406 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T13:49:25.652 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T13:49:26.686 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T13:49:26.721 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:49:29.364 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 1/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T13:49:29.365 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T13:49:29.807 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T13:49:29.808 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T13:49:29.809 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:29.809 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:30.217 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-rook 2024-03-30T13:49:30.791 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:33 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:30.945 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 49 k 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T13:49:30.947 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout:Install 13 Packages 2024-03-30T13:49:30.948 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:30.949 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 2.0 M 2024-03-30T13:49:30.949 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 26 M 2024-03-30T13:49:30.949 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:31.554 INFO:teuthology.orchestra.run.smithi138.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-30T13:49:31.579 INFO:teuthology.orchestra.run.smithi138.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 133 kB/s | 24 kB 00:00 2024-03-30T13:49:31.630 INFO:teuthology.orchestra.run.smithi138.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-30T13:49:31.655 INFO:teuthology.orchestra.run.smithi138.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-30T13:49:31.680 INFO:teuthology.orchestra.run.smithi138.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-30T13:49:31.706 INFO:teuthology.orchestra.run.smithi138.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 852 kB/s | 43 kB 00:00 2024-03-30T13:49:31.731 INFO:teuthology.orchestra.run.smithi138.stdout:(7/13): ceph-mgr-rook-19.0.0-1967.g4e50bbcd.el9 147 kB/s | 49 kB 00:00 2024-03-30T13:49:31.849 INFO:teuthology.orchestra.run.smithi138.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-30T13:49:31.874 INFO:teuthology.orchestra.run.smithi138.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 89 kB/s | 15 kB 00:00 2024-03-30T13:49:31.966 INFO:teuthology.orchestra.run.smithi138.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 646 kB/s | 59 kB 00:00 2024-03-30T13:49:31.991 INFO:teuthology.orchestra.run.smithi138.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 956 kB/s | 203 kB 00:00 2024-03-30T13:49:32.033 INFO:teuthology.orchestra.run.smithi138.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2024-03-30T13:49:32.058 INFO:teuthology.orchestra.run.smithi138.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 982 kB/s | 90 kB 00:00 2024-03-30T13:49:32.062 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:32.063 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-30T13:49:32.063 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:32.076 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:32.076 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:32.168 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:32.169 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:32.687 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:32.847 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T13:49:32.892 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T13:49:32.957 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T13:49:32.995 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T13:49:33.035 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T13:49:33.130 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T13:49:33.238 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T13:49:33.288 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T13:49:34.007 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T13:49:34.053 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T13:49:34.086 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T13:49:34.135 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T13:49:34.165 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T13:49:34.900 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T13:49:34.901 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T13:49:34.901 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T13:49:34.901 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T13:49:34.901 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T13:49:35.250 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T13:49:35.250 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:35.250 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:35.250 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:49:35.250 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:35.251 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:35.518 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T13:49:36.082 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:36.237 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:36.237 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-30T13:49:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:49:36.237 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-30T13:49:36.237 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 139 k 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================== 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-03-30T13:49:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:36.239 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 6.8 M 2024-03-30T13:49:36.239 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 33 M 2024-03-30T13:49:36.239 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:36.732 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1967.g4e50bbcd.e 602 kB/s | 139 kB 00:00 2024-03-30T13:49:36.865 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-30T13:49:37.040 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 332 kB/s | 58 kB 00:00 2024-03-30T13:49:37.066 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-30T13:49:37.116 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 677 kB/s | 51 kB 00:00 2024-03-30T13:49:37.466 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 257 kB/s | 248 kB 00:00 2024-03-30T13:49:37.468 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:37.468 INFO:teuthology.orchestra.run.smithi138.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-30T13:49:37.483 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:37.494 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:37.494 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:37.585 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:37.587 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:37.877 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:38.040 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T13:49:38.093 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T13:49:38.609 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T13:49:38.689 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T13:49:38.758 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T13:49:38.782 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T13:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/6 2024-03-30T13:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T13:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T13:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T13:49:39.636 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T13:49:39.939 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:39.940 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:40.216 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-fuse 2024-03-30T13:49:40.780 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:40.932 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 889 k 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:40.933 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-03-30T13:49:40.934 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:40.934 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 968 k 2024-03-30T13:49:40.934 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 2.8 M 2024-03-30T13:49:40.934 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:41.179 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 322 kB/s | 79 kB 00:00 2024-03-30T13:49:42.096 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-fuse-19.0.0-1967.g4e50bbcd.el9.x86_ 766 kB/s | 889 kB 00:01 2024-03-30T13:49:42.097 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:42.098 INFO:teuthology.orchestra.run.smithi138.stdout:Total 832 kB/s | 968 kB 00:01 2024-03-30T13:49:42.098 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:42.122 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:42.123 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:42.201 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:42.201 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:42.407 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:42.547 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T13:49:42.613 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:49:43.283 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T13:49:43.283 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/2 2024-03-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:43.881 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-volume 2024-03-30T13:49:44.465 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:47 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:44.616 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 264 k 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:44.617 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:44.618 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-03-30T13:49:44.618 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:44.618 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 332 k 2024-03-30T13:49:44.618 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.5 M 2024-03-30T13:49:44.618 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:44.820 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 340 kB/s | 69 kB 00:00 2024-03-30T13:49:44.862 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-volume-19.0.0-1967.g4e50bbcd.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-30T13:49:44.863 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:44.864 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-30T13:49:44.864 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:44.873 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:44.873 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:44.907 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:44.908 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:45.060 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:45.253 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T13:49:45.291 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:49:45.307 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T13:49:45.308 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:49:45.308 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T13:49:45.308 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:46.004 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/2 2024-03-30T13:49:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T13:49:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:46.332 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T13:49:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T13:49:46.332 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:46.332 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:46.658 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados-devel 2024-03-30T13:49:47.239 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:47.391 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 127 k 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-30T13:49:47.392 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:47.393 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 127 k 2024-03-30T13:49:47.393 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 456 k 2024-03-30T13:49:47.393 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout:librados-devel-19.0.0-1967.g4e50bbcd.el9.x86_64 623 kB/s | 127 kB 00:00 2024-03-30T13:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout:Total 619 kB/s | 127 kB 00:00 2024-03-30T13:49:47.597 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:47.603 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:47.603 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:47.658 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:47.659 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:47.814 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:47.904 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:48.377 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:48.672 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:49:48.672 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:48.673 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:48.917 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs2 2024-03-30T13:49:49.494 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:52 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:49.596 INFO:teuthology.orchestra.run.smithi138.stdout:Package libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:49.650 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:49.651 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:49.652 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:49.716 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs-devel 2024-03-30T13:49:50.285 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:53 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:50.437 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 32 k 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:50.438 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 32 k 2024-03-30T13:49:50.439 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 142 k 2024-03-30T13:49:50.439 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:50.612 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-devel-19.0.0-1967.g4e50bbcd.el9.x86_6 183 kB/s | 32 kB 00:00 2024-03-30T13:49:50.612 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:50.612 INFO:teuthology.orchestra.run.smithi138.stdout:Total 182 kB/s | 32 kB 00:00 2024-03-30T13:49:50.612 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:50.616 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:50.616 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:50.643 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:50.644 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:50.779 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:50.885 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:51.261 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:51.512 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:49:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:51.512 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:51.707 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados2 2024-03-30T13:49:52.270 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:55 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:52.368 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:52.420 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:52.421 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:52.421 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:52.475 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librbd1 2024-03-30T13:49:53.045 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:56 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:53.145 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:53.196 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:53.197 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:53.197 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:53.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rados 2024-03-30T13:49:53.808 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:56 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:53.907 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:53.958 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:53.959 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:53.960 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:54.004 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rgw 2024-03-30T13:49:54.568 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:57 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:54.667 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:54.718 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:54.719 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:54.720 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:54.764 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-cephfs 2024-03-30T13:49:55.331 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:58 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:55.430 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:55.483 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:55.484 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:55.484 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:55.529 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rbd 2024-03-30T13:49:56.104 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:59 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:56.202 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T13:49:56.253 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:56.254 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-03-30T13:49:56.254 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:56.298 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-fuse 2024-03-30T13:49:56.866 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:59 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:57.033 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:57.034 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:57.034 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-30T13:49:57.034 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:57.034 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 85 k 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 85 k 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 231 k 2024-03-30T13:49:57.035 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:57.416 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 224 kB/s | 85 kB 00:00 2024-03-30T13:49:57.417 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:57.417 INFO:teuthology.orchestra.run.smithi138.stdout:Total 223 kB/s | 85 kB 00:00 2024-03-30T13:49:57.417 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:57.425 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:57.425 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:57.481 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:57.482 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:49:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:49:57.724 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:58.150 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:58.458 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:49:58.458 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:58.458 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:49:58.458 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:49:58.459 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:58.459 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:49:58.677 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-mirror 2024-03-30T13:49:59.256 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:02 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:49:59.413 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:49:59.414 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.1 M 2024-03-30T13:49:59.415 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 13 M 2024-03-30T13:49:59.415 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:49:59.806 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-mirror-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2024-03-30T13:49:59.806 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:49:59.807 INFO:teuthology.orchestra.run.smithi138.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-03-30T13:49:59.807 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:49:59.816 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:49:59.816 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:49:59.889 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:49:59.890 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:50:00.315 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:50:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:50:00.431 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:50:00.431 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T13:50:00.431 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T13:50:00.432 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:50:00.432 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T13:50:00.432 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:50:01.548 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:50:01.548 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:50:01.549 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:50:01.549 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:50:01.549 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:50:01.549 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:50:01.773 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-nbd 2024-03-30T13:50:02.353 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:02:05 ago on Sat 30 Mar 2024 01:47:57 PM UTC. 2024-03-30T13:50:02.506 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 170 k 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 170 k 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 494 k 2024-03-30T13:50:02.507 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-03-30T13:50:02.854 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-nbd-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 492 kB/s | 170 kB 00:00 2024-03-30T13:50:02.855 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:50:02.855 INFO:teuthology.orchestra.run.smithi138.stdout:Total 490 kB/s | 170 kB 00:00 2024-03-30T13:50:02.855 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-03-30T13:50:02.862 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-03-30T13:50:02.863 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-03-30T13:50:02.920 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-03-30T13:50:02.920 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-03-30T13:50:03.072 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-03-30T13:50:03.168 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:50:03.639 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:50:03.925 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T13:50:03.925 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:50:03.925 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-03-30T13:50:03.925 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T13:50:03.925 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:50:03.926 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-03-30T13:50:04.170 DEBUG:teuthology.parallel:result is None 2024-03-30T13:50:04.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:50:04.314 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T13:50:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:50:04.341 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:50:04.341 INFO:teuthology.task.install:The correct ceph version 19.0.0-1967.g4e50bbcd is installed. 2024-03-30T13:50:04.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:50:04.557 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T13:50:04.578 INFO:teuthology.orchestra.run.smithi138.stdout:19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:50:04.578 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1967.g4e50bbcd.el9 2024-03-30T13:50:04.578 INFO:teuthology.task.install:The correct ceph version 19.0.0-1967.g4e50bbcd is installed. 2024-03-30T13:50:04.580 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T13:50:04.581 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:50:04.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T13:50:04.615 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:50:04.615 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T13:50:04.647 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T13:50:04.647 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:50:04.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T13:50:04.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T13:50:04.761 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:50:04.761 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T13:50:04.796 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T13:50:04.866 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T13:50:04.866 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:50:04.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T13:50:04.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T13:50:04.969 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:50:04.969 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T13:50:04.995 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T13:50:05.061 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T13:50:05.062 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:50:05.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T13:50:05.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T13:50:05.158 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:50:05.158 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T13:50:05.185 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T13:50:05.252 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-30T13:50:05.348 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'cephadm_mode': 'cephadm-package'} 2024-03-30T13:50:05.349 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:50:05.349 INFO:tasks.cephadm:Cluster fsid is 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:50:05.349 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-30T13:50:05.350 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.102', 'mon.c': '[v2:172.21.15.102:3301,v1:172.21.15.102:6790]', 'mon.b': '172.21.15.138'} 2024-03-30T13:50:05.350 INFO:tasks.cephadm:First mon is mon.a on smithi102 2024-03-30T13:50:05.350 INFO:tasks.cephadm:First mgr is y 2024-03-30T13:50:05.350 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-30T13:50:05.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-03-30T13:50:05.377 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname $(hostname -s) 2024-03-30T13:50:05.415 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb on all hosts... 2024-03-30T13:50:05.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb pull 2024-03-30T13:50:05.419 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb pull 2024-03-30T13:50:06.225 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:50:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:50:37.907 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-03-30T13:50:37.907 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T13:50:37.907 INFO:teuthology.orchestra.run.smithi138.stdout: "image_id": "b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e", 2024-03-30T13:50:37.907 INFO:teuthology.orchestra.run.smithi138.stdout: "repo_digests": [ 2024-03-30T13:50:37.907 INFO:teuthology.orchestra.run.smithi138.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d" 2024-03-30T13:50:37.907 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-03-30T13:50:37.907 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-03-30T13:50:51.654 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-30T13:50:51.654 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T13:50:51.654 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e", 2024-03-30T13:50:51.655 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-03-30T13:50:51.655 INFO:teuthology.orchestra.run.smithi102.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d" 2024-03-30T13:50:51.655 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-30T13:50:51.655 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-30T13:50:51.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-03-30T13:50:51.742 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph 2024-03-30T13:50:51.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-03-30T13:50:51.805 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /etc/ceph 2024-03-30T13:50:51.848 INFO:tasks.cephadm:Writing seed config... 2024-03-30T13:50:51.849 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-30T13:50:51.849 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-30T13:50:51.849 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-30T13:50:51.849 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-30T13:50:51.849 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-30T13:50:51.849 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-30T13:50:51.850 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-30T13:50:51.851 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:50:51.851 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-30T13:50:51.866 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 mon election default strategy = 1 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-30T13:50:51.866 DEBUG:teuthology.orchestra.run.smithi102:mon.a> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service 2024-03-30T13:50:51.910 DEBUG:teuthology.orchestra.run.smithi102:mgr.y> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.y.service 2024-03-30T13:50:51.953 INFO:tasks.cephadm:Bootstrapping... 2024-03-30T13:50:51.953 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -v bootstrap --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.102 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-30T13:50:52.145 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-30T13:50:52.145 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', '-v', 'bootstrap', '--fsid', '69dd7ec8-ee9c-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.102', '--skip-admin-label'] 2024-03-30T13:50:52.145 INFO:teuthology.orchestra.run.smithi102.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-30T13:50:52.146 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying podman|docker is present... 2024-03-30T13:50:52.170 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T13:50:52.171 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying lvm2 is present... 2024-03-30T13:50:52.171 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying time synchronization is in place... 2024-03-30T13:50:52.178 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T13:50:52.178 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T13:50:52.184 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T13:50:52.184 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-30T13:50:52.190 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-03-30T13:50:52.196 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-03-30T13:50:52.196 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-03-30T13:50:52.196 INFO:teuthology.orchestra.run.smithi102.stdout:Repeating the final host check... 2024-03-30T13:50:52.218 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T13:50:52.219 INFO:teuthology.orchestra.run.smithi102.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-30T13:50:52.219 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl is present 2024-03-30T13:50:52.219 INFO:teuthology.orchestra.run.smithi102.stdout:lvcreate is present 2024-03-30T13:50:52.225 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T13:50:52.225 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T13:50:52.231 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T13:50:52.231 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-30T13:50:52.237 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-03-30T13:50:52.243 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-03-30T13:50:52.243 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-03-30T13:50:52.243 INFO:teuthology.orchestra.run.smithi102.stdout:Host looks OK 2024-03-30T13:50:52.243 INFO:teuthology.orchestra.run.smithi102.stdout:Cluster fsid: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:50:52.243 INFO:teuthology.orchestra.run.smithi102.stdout:Acquiring lock 140528762174240 on /run/cephadm/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.lock 2024-03-30T13:50:52.243 INFO:teuthology.orchestra.run.smithi102.stdout:Lock 140528762174240 acquired on /run/cephadm/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.lock 2024-03-30T13:50:52.244 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 3300 ... 2024-03-30T13:50:52.244 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 6789 ... 2024-03-30T13:50:52.244 INFO:teuthology.orchestra.run.smithi102.stdout:Base mon IP(s) is [172.21.15.102:3300, 172.21.15.102:6789], mon addrv is [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-03-30T13:50:52.247 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.102 metric 100 2024-03-30T13:50:52.247 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.102 metric 100 2024-03-30T13:50:52.249 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-30T13:50:52.249 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-30T13:50:52.250 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-03-30T13:50:52.252 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-30T13:50:52.252 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-30T13:50:52.252 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T13:50:52.252 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-30T13:50:52.252 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-30T13:50:52.252 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T13:50:52.253 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-03-30T13:50:52.253 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-03-30T13:50:52.253 INFO:teuthology.orchestra.run.smithi102.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-30T13:50:52.253 INFO:teuthology.orchestra.run.smithi102.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-30T13:50:52.254 INFO:teuthology.orchestra.run.smithi102.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:50:52.831 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e 2024-03-30T13:50:52.831 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T13:50:52.831 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Getting image source signatures 2024-03-30T13:50:52.831 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:70aeeb6837442a4655a911c520b59d95aba287a7bb7c28348ac976aba07eb7c4 2024-03-30T13:50:52.831 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-30T13:50:52.831 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying config sha256:b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e 2024-03-30T13:50:52.831 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-30T13:50:53.786 INFO:teuthology.orchestra.run.smithi102.stdout:ceph: stdout ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev) 2024-03-30T13:50:53.786 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph version: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev) 2024-03-30T13:50:53.787 INFO:teuthology.orchestra.run.smithi102.stdout:Extracting ceph user uid/gid from container image... 2024-03-30T13:50:56.359 INFO:teuthology.orchestra.run.smithi102.stdout:stat: stdout 167 167 2024-03-30T13:50:56.359 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial keys... 2024-03-30T13:51:05.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQDDGAhmDWR3GxAAwrwmqWXIydYmkV0An4lrwA== 2024-03-30T13:51:08.744 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQDMGAhmEHRrAhAAx3P4RYDL8achJuvAkF4mfQ== 2024-03-30T13:51:09.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQDNGAhmWtIwBBAAe2iqqWD/oK0mQQNkdro+5Q== 2024-03-30T13:51:09.567 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial monmap... 2024-03-30T13:51:10.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool for a [v2:172.21.15.102:3300,v1:172.21.15.102:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = quincy 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: set fsid to 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:51:10.548 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mon... 2024-03-30T13:51:11.445 INFO:teuthology.orchestra.run.smithi102.stdout:create mon.a on 2024-03-30T13:51:11.663 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-30T13:51:11.806 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-30T13:51:12.002 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.target. 2024-03-30T13:51:12.002 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.target. 2024-03-30T13:51:12.221 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T13:51:12.222 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-03-30T13:51:12.401 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.target.wants/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@.service. 2024-03-30T13:51:12.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:12 smithi102 podman[29001]: 2024-03-30 13:51:12.542505445 +0000 UTC m=+0.017434442 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:51:12.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:12 smithi102 podman[29001]: 2024-03-30 13:51:12.92853251 +0000 UTC m=+0.403461540 container create 809fe3ad6651cb230c941b4f69e03c77e1952bade18b7cceb430c25129bd9946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-30T13:51:13.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:13 smithi102 podman[29001]: 2024-03-30 13:51:13.097760968 +0000 UTC m=+0.572689954 container init 809fe3ad6651cb230c941b4f69e03c77e1952bade18b7cceb430c25129bd9946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-4e50bbc) 2024-03-30T13:51:13.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:13 smithi102 podman[29001]: 2024-03-30 13:51:13.101847021 +0000 UTC m=+0.576776013 container start 809fe3ad6651cb230c941b4f69e03c77e1952bade18b7cceb430c25129bd9946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-30T13:51:13.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:13 smithi102 ceph-mon[29020]: mkfs 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:13.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:13 smithi102 ceph-mon[29020]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T13:51:13.929 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:51:13.930 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-30T13:51:13.937 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:51:13.937 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-30T13:51:13.937 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-30T13:51:13.937 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T13:51:13.938 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon to start... 2024-03-30T13:51:13.938 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon... 2024-03-30T13:51:14.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:13 smithi102 bash[29001]: 809fe3ad6651cb230c941b4f69e03c77e1952bade18b7cceb430c25129bd9946 2024-03-30T13:51:14.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:13 smithi102 systemd[1]: Started Ceph mon.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:51:14.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: monmap epoch 1 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: last_changed 2024-03-30T13:51:09.933456+0000 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: created 2024-03-30T13:51:09.933456+0000 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: min_mon_release 19 (squid) 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: election_strategy: 1 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: fsmap 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T13:51:14.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:14 smithi102 ceph-mon[29020]: mgrmap e1: no daemons active 2024-03-30T13:51:15.021 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout cluster: 2024-03-30T13:51:15.021 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout id: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:15.021 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-30T13:51:15.021 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:15.021 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout services: 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 1.31738s) 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout data: 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pgs: 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.382+0000 7f95790be700 1 Processor -- start 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.382+0000 7f95790be700 1 -- start start 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.383+0000 7f95790be700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957407a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:15.022 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.383+0000 7f95790be700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f957407aa30 con 0x7f957407c000 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.383+0000 7f9572d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.383+0000 7f9572d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957407a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50144/0 (socket says 172.21.15.102:50144) 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.383+0000 7f9572d9d700 1 -- 172.21.15.102:0/2236465592 learned_addr learned my addr 172.21.15.102:0/2236465592 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.383+0000 7f9572d9d700 1 -- 172.21.15.102:0/2236465592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957407abb0 con 0x7f957407c000 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.383+0000 7f9572d9d700 1 --2- 172.21.15.102:0/2236465592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957407a460 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95680098a0 tx=0x7f9568009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1557db3e243ca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f9571d9b700 1 -- 172.21.15.102:0/2236465592 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9568004160 con 0x7f957407c000 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f9571d9b700 1 -- 172.21.15.102:0/2236465592 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9568004300 con 0x7f957407c000 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f9571d9b700 1 -- 172.21.15.102:0/2236465592 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95680036a0 con 0x7f957407c000 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f95790be700 1 -- 172.21.15.102:0/2236465592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 msgr2=0x7f957407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f95790be700 1 --2- 172.21.15.102:0/2236465592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957407a460 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95680098a0 tx=0x7f9568009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f95790be700 1 -- 172.21.15.102:0/2236465592 shutdown_connections 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f95790be700 1 --2- 172.21.15.102:0/2236465592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957407a460 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f95790be700 1 -- 172.21.15.102:0/2236465592 >> 172.21.15.102:0/2236465592 conn(0x7f9574101800 msgr2=0x7f9574103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:15.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f95790be700 1 -- 172.21.15.102:0/2236465592 shutdown_connections 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.384+0000 7f95790be700 1 -- 172.21.15.102:0/2236465592 wait complete. 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.385+0000 7f95790be700 1 Processor -- start 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.385+0000 7f95790be700 1 -- start start 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.385+0000 7f95790be700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.385+0000 7f95790be700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f957407b0e0 con 0x7f957407c000 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f9572d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f9572d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957419bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50158/0 (socket says 172.21.15.102:50158) 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f9572d9d700 1 -- 172.21.15.102:0/1953366559 learned_addr learned my addr 172.21.15.102:0/1953366559 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f9572d9d700 1 -- 172.21.15.102:0/1953366559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957419c100 con 0x7f957407c000 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f9572d9d700 1 --2- 172.21.15.102:0/1953366559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957419bbc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9568000c00 tx=0x7f9568004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f955bfff700 1 -- 172.21.15.102:0/1953366559 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9568003db0 con 0x7f957407c000 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f955bfff700 1 -- 172.21.15.102:0/1953366559 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9568011630 con 0x7f957407c000 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957419c390 con 0x7f957407c000 2024-03-30T13:51:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f955bfff700 1 -- 172.21.15.102:0/1953366559 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9568011950 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f957419f080 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f955bfff700 1 -- 172.21.15.102:0/1953366559 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f956801a460 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.386+0000 7f955bfff700 1 -- 172.21.15.102:0/1953366559 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f956801b780 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.387+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9574063870 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.389+0000 7f955bfff700 1 -- 172.21.15.102:0/1953366559 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9568031020 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.460+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f957402db30 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.460+0000 7f955bfff700 1 -- 172.21.15.102:0/1953366559 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f956801b9d0 con 0x7f957407c000 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.461+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 msgr2=0x7f957419bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.461+0000 7f95790be700 1 --2- 172.21.15.102:0/1953366559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957419bbc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9568000c00 tx=0x7f9568004750 comp rx=0 tx=0).stop 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.461+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 shutdown_connections 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.461+0000 7f95790be700 1 --2- 172.21.15.102:0/1953366559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f957407c000 0x7f957419bbc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.461+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 >> 172.21.15.102:0/1953366559 conn(0x7f9574101800 msgr2=0x7f9574101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.461+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 shutdown_connections 2024-03-30T13:51:15.025 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:14.461+0000 7f95790be700 1 -- 172.21.15.102:0/1953366559 wait complete. 2024-03-30T13:51:15.026 INFO:teuthology.orchestra.run.smithi102.stdout:mon is available 2024-03-30T13:51:15.026 INFO:teuthology.orchestra.run.smithi102.stdout:Assimilating anything we can from ceph.conf... 2024-03-30T13:51:15.264 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:15 smithi102 ceph-mon[29020]: from='client.? 172.21.15.102:0/1953366559' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-30T13:51:16.094 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:16.094 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T13:51:16.094 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:16.094 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f0260ae0700 1 Processor -- start 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f0260ae0700 1 -- start start 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f0260ae0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f0260ae0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f025c106c50 con 0x7f025c106260 2024-03-30T13:51:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f025a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f025a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50162/0 (socket says 172.21.15.102:50162) 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f025a59c700 1 -- 172.21.15.102:0/482192854 learned_addr learned my addr 172.21.15.102:0/482192854 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.438+0000 7f025a59c700 1 -- 172.21.15.102:0/482192854 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025c107460 con 0x7f025c106260 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f025a59c700 1 --2- 172.21.15.102:0/482192854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c106680 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f024c0098a0 tx=0x7f024c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7174207338f224 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f025959a700 1 -- 172.21.15.102:0/482192854 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f024c004160 con 0x7f025c106260 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f025959a700 1 -- 172.21.15.102:0/482192854 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f024c004300 con 0x7f025c106260 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f025959a700 1 -- 172.21.15.102:0/482192854 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f024c0036a0 con 0x7f025c106260 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f0260ae0700 1 -- 172.21.15.102:0/482192854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 msgr2=0x7f025c106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f0260ae0700 1 --2- 172.21.15.102:0/482192854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c106680 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f024c0098a0 tx=0x7f024c009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f0260ae0700 1 -- 172.21.15.102:0/482192854 shutdown_connections 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f0260ae0700 1 --2- 172.21.15.102:0/482192854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c106680 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.439+0000 7f0260ae0700 1 -- 172.21.15.102:0/482192854 >> 172.21.15.102:0/482192854 conn(0x7f025c101800 msgr2=0x7f025c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.440+0000 7f0260ae0700 1 -- 172.21.15.102:0/482192854 shutdown_connections 2024-03-30T13:51:16.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.440+0000 7f0260ae0700 1 -- 172.21.15.102:0/482192854 wait complete. 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.440+0000 7f0260ae0700 1 Processor -- start 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.440+0000 7f0260ae0700 1 -- start start 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.440+0000 7f0260ae0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.440+0000 7f0260ae0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f025c107990 con 0x7f025c106260 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f025a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f025a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c1999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50176/0 (socket says 172.21.15.102:50176) 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f025a59c700 1 -- 172.21.15.102:0/273090641 learned_addr learned my addr 172.21.15.102:0/273090641 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f025a59c700 1 -- 172.21.15.102:0/273090641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025c199f30 con 0x7f025c106260 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f025a59c700 1 --2- 172.21.15.102:0/273090641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c1999f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f024c000c00 tx=0x7f024c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f024c003db0 con 0x7f025c106260 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f024c011630 con 0x7f025c106260 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f024c011950 con 0x7f025c106260 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f025c19a1c0 con 0x7f025c106260 2024-03-30T13:51:16.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f025c19ceb0 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f024c01a460 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.441+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f024c01b780 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.442+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f025c063870 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.444+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f024c031020 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.512+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f025c19a6a0 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.515+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+340 (secure 0 0 0) 0x7f025c19a6a0 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.515+0000 7f024b7fe700 1 -- 172.21.15.102:0/273090641 <== mon.0 v2:172.21.15.102:3300/0 8 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f024c01a8e0 con 0x7f025c106260 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.516+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 msgr2=0x7f025c1999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.516+0000 7f0260ae0700 1 --2- 172.21.15.102:0/273090641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c1999f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f024c000c00 tx=0x7f024c004750 comp rx=0 tx=0).stop 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.516+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 shutdown_connections 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.516+0000 7f0260ae0700 1 --2- 172.21.15.102:0/273090641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c106260 0x7f025c1999f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.516+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 >> 172.21.15.102:0/273090641 conn(0x7f025c101800 msgr2=0x7f025c1905b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:16.098 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.516+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 shutdown_connections 2024-03-30T13:51:16.099 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:15.516+0000 7f0260ae0700 1 -- 172.21.15.102:0/273090641 wait complete. 2024-03-30T13:51:16.099 INFO:teuthology.orchestra.run.smithi102.stdout:Generating new minimal ceph.conf... 2024-03-30T13:51:16.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:16 smithi102 ceph-mon[29020]: from='client.? 172.21.15.102:0/273090641' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T13:51:16.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:16 smithi102 ceph-mon[29020]: from='client.? 172.21.15.102:0/273090641' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-30T13:51:17.140 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.474+0000 7f79db182700 1 Processor -- start 2024-03-30T13:51:17.140 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79db182700 1 -- start start 2024-03-30T13:51:17.140 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79db182700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d4106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:17.140 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79db182700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4106c50 con 0x7f79d4106260 2024-03-30T13:51:17.140 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79d8f1e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d4106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:17.140 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79d8f1e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d4106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50188/0 (socket says 172.21.15.102:50188) 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79d8f1e700 1 -- 172.21.15.102:0/3797938445 learned_addr learned my addr 172.21.15.102:0/3797938445 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79d8f1e700 1 -- 172.21.15.102:0/3797938445 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d4107460 con 0x7f79d4106260 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79d8f1e700 1 --2- 172.21.15.102:0/3797938445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d4106680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098a0 tx=0x7f79c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702890462d8f4186 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.475+0000 7f79d37fe700 1 -- 172.21.15.102:0/3797938445 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c8004160 con 0x7f79d4106260 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79d37fe700 1 -- 172.21.15.102:0/3797938445 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f79c8004300 con 0x7f79d4106260 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79d37fe700 1 -- 172.21.15.102:0/3797938445 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c8011a60 con 0x7f79d4106260 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79db182700 1 -- 172.21.15.102:0/3797938445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 msgr2=0x7f79d4106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79db182700 1 --2- 172.21.15.102:0/3797938445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d4106680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098a0 tx=0x7f79c8009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79db182700 1 -- 172.21.15.102:0/3797938445 shutdown_connections 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79db182700 1 --2- 172.21.15.102:0/3797938445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d4106680 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79db182700 1 -- 172.21.15.102:0/3797938445 >> 172.21.15.102:0/3797938445 conn(0x7f79d4101800 msgr2=0x7f79d4103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:17.141 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79db182700 1 -- 172.21.15.102:0/3797938445 shutdown_connections 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.476+0000 7f79db182700 1 -- 172.21.15.102:0/3797938445 wait complete. 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79db182700 1 Processor -- start 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79db182700 1 -- start start 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79db182700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79db182700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f79d410a810 con 0x7f79d4106260 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79d8f1e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79d8f1e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50196/0 (socket says 172.21.15.102:50196) 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79d8f1e700 1 -- 172.21.15.102:0/786592991 learned_addr learned my addr 172.21.15.102:0/786592991 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79d8f1e700 1 -- 172.21.15.102:0/786592991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d419a730 con 0x7f79d4106260 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79d8f1e700 1 --2- 172.21.15.102:0/786592991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d419a1f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009fb0 tx=0x7f79c8011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.477+0000 7f79d1ffb700 1 -- 172.21.15.102:0/786592991 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c80039e0 con 0x7f79d4106260 2024-03-30T13:51:17.142 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.478+0000 7f79d1ffb700 1 -- 172.21.15.102:0/786592991 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f79c801a8c0 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.478+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d419a9c0 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.478+0000 7f79d1ffb700 1 -- 172.21.15.102:0/786592991 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c8024800 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.478+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d419d6b0 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.478+0000 7f79d1ffb700 1 -- 172.21.15.102:0/786592991 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f79c8022020 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.478+0000 7f79d1ffb700 1 -- 172.21.15.102:0/786592991 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f79c80193c0 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.479+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d4063870 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.481+0000 7f79d1ffb700 1 -- 172.21.15.102:0/786592991 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f79c801aa70 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.547+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f79d419da00 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.548+0000 7f79d1ffb700 1 -- 172.21.15.102:0/786592991 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f79c801a430 con 0x7f79d4106260 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.548+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 msgr2=0x7f79d419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.548+0000 7f79db182700 1 --2- 172.21.15.102:0/786592991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d419a1f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009fb0 tx=0x7f79c8011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:17.143 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.548+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 shutdown_connections 2024-03-30T13:51:17.144 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.548+0000 7f79db182700 1 --2- 172.21.15.102:0/786592991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79d4106260 0x7f79d419a1f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:17.144 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.548+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 >> 172.21.15.102:0/786592991 conn(0x7f79d4101800 msgr2=0x7f79d41022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:17.144 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.549+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 shutdown_connections 2024-03-30T13:51:17.144 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:16.549+0000 7f79db182700 1 -- 172.21.15.102:0/786592991 wait complete. 2024-03-30T13:51:17.144 INFO:teuthology.orchestra.run.smithi102.stdout:Restarting the monitor... 2024-03-30T13:51:17.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:17 smithi102 systemd[1]: Stopping Ceph mon.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:51:17.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:17 smithi102 ceph-mon[29020]: from='client.? 172.21.15.102:0/786592991' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:51:17.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a[29016]: 2024-03-30T13:51:17.377+0000 7ff669fe7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T13:51:17.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a[29016]: 2024-03-30T13:51:17.377+0000 7ff669fe7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T13:51:17.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:17 smithi102 podman[29264]: 2024-03-30 13:51:17.521147281 +0000 UTC m=+0.228042218 container died 809fe3ad6651cb230c941b4f69e03c77e1952bade18b7cceb430c25129bd9946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T13:51:18.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:17 smithi102 podman[29264]: 2024-03-30 13:51:17.911084117 +0000 UTC m=+0.617979062 container cleanup 809fe3ad6651cb230c941b4f69e03c77e1952bade18b7cceb430c25129bd9946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-30T13:51:18.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:17 smithi102 bash[29264]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T13:51:18.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 podman[29278]: 2024-03-30 13:51:18.153674161 +0000 UTC m=+0.628065507 container remove 809fe3ad6651cb230c941b4f69e03c77e1952bade18b7cceb430c25129bd9946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T13:51:18.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-03-30T13:51:18.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 systemd[1]: Stopped Ceph mon.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:51:18.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 systemd[1]: Starting Ceph mon.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:51:18.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 podman[29373]: 2024-03-30 13:51:18.393408751 +0000 UTC m=+0.018486845 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:51:18.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 podman[29373]: 2024-03-30 13:51:18.496475081 +0000 UTC m=+0.121553142 container create 5062081f6df6a13272035cb1b3c6ad7aebc7b35cd284325315c6b521d11c60e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-30T13:51:18.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 podman[29373]: 2024-03-30 13:51:18.691053578 +0000 UTC m=+0.316131642 container init 5062081f6df6a13272035cb1b3c6ad7aebc7b35cd284325315c6b521d11c60e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 podman[29373]: 2024-03-30 13:51:18.694310188 +0000 UTC m=+0.319388253 container start 5062081f6df6a13272035cb1b3c6ad7aebc7b35cd284325315c6b521d11c60e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux ) 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 2 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: pidfile_write: ignore empty --pid-file 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: load: jerasure load: lrc 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: RocksDB version: 7.9.2 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Git sha 0 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: DB SUMMARY 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: DB Session ID: TADZ8FGYK67OJ9FA3A08 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: CURRENT file: CURRENT 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-30T13:51:18.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92158 ; 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.error_if_exists: 0 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.create_if_missing: 0 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.paranoid_checks: 1 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.env: 0x5581f84f9c60 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.info_log: 0x5581fa201ae0 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.statistics: (nil) 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.use_fsync: 0 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_log_file_size: 0 2024-03-30T13:51:18.776 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.allow_fallocate: 1 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.use_direct_reads: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.db_log_dir: 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.wal_dir: 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T13:51:18.777 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.write_buffer_manager: 0x5581fa1d41e0 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T13:51:18.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T13:51:18.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.unordered_write: 0 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.row_cache: None 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.wal_filter: None 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.two_write_queues: 0 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T13:51:18.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.wal_compression: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.atomic_flush: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.log_readahead_size: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_background_jobs: 2 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_background_compactions: -1 2024-03-30T13:51:18.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_subcompactions: 1 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_open_files: -1 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T13:51:18.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_background_flushes: -1 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Compression algorithms supported: 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kZSTD supported: 0 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kXpressCompression supported: 0 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kZlibCompression supported: 1 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kSnappyCompression supported: 1 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kLZ4Compression supported: 1 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: kBZip2Compression supported: 0 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T13:51:18.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.merge_operator: 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_filter: None 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5581fa201be0) 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_type: 0 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_index_type: 0 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_shortening: 1 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T13:51:18.784 INFO:journalctl@ceph.mon.a.smithi102.stdout: checksum: 4 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: no_block_cache: 0 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache: 0x5581fa205090 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_options: 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: capacity : 536870912 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_shard_bits : 4 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: strict_capacity_limit : 0 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_compressed: (nil) 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: persistent_cache: (nil) 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size: 4096 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size_deviation: 10 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_restart_interval: 16 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_block_restart_interval: 1 2024-03-30T13:51:18.785 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_block_size: 4096 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: partition_filters: 0 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: use_delta_encoding: 1 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: filter_policy: bloomfilter 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: whole_key_filtering: 1 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: verify_compression: 0 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: format_version: 5 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_index_compression: 1 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_align: 0 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_auto_readahead_size: 262144 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: prepopulate_block_cache: 0 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: initial_auto_readahead_size: 8192 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T13:51:18.786 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression: NoCompression 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.num_levels: 7 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:51:18.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T13:51:18.788 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T13:51:18.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T13:51:18.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.table_properties_collectors: 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.inplace_update_support: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.bloom_locality: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.max_successive_merges: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.ttl: 2592000 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T13:51:18.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.enable_blob_files: false 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.min_blob_size: 0 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 76639016-f29f-41bb-8f15-b10ff90a7858 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711806678723026, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-30T13:51:18.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711806678723887, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88325, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 86408, "index_size": 263, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11018, "raw_average_key_size": 47, "raw_value_size": 80238, "raw_average_value_size": 344, "num_data_blocks": 12, "num_entries": 233, "num_filter_entries": 233, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711806678, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "76639016-f29f-41bb-8f15-b10ff90a7858", "db_session_id": "TADZ8FGYK67OJ9FA3A08", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711806678723971, "job": 1, "event": "recovery_finished"} 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5581fa2f6000 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: DB pointer 0x5581fa2e0000 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** DB Stats ** 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:51:18.793 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: L0 2/0 88.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: Sum 2/0 88.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T13:51:18.794 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative compaction: 0.00 GB write, 30.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval compaction: 0.00 GB write, 30.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache BinnedLRUCache@0x5581fa205090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-30T13:51:18.795 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: starting mon.a rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???) e1 preinit fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???).mds e1 new map 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???).mds e1 print_map 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout: e1 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout: legacy client fscid: -1 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout: No filesystems configured 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:51:18.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-30T13:51:18.797 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 bash[29373]: 5062081f6df6a13272035cb1b3c6ad7aebc7b35cd284325315c6b521d11c60e2 2024-03-30T13:51:18.797 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 systemd[1]: Started Ceph mon.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:51:18.797 INFO:teuthology.orchestra.run.smithi102.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: monmap epoch 1 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: last_changed 2024-03-30T13:51:09.933456+0000 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: created 2024-03-30T13:51:09.933456+0000 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: min_mon_release 19 (squid) 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: election_strategy: 1 2024-03-30T13:51:19.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-30T13:51:19.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: fsmap 2024-03-30T13:51:19.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T13:51:19.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:18 smithi102 ceph-mon[29392]: mgrmap e1: no daemons active 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.187+0000 7f4a78de0700 1 Processor -- start 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.187+0000 7f4a78de0700 1 -- start start 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.187+0000 7f4a78de0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a741090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.187+0000 7f4a78de0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74079f90 con 0x7f4a74106d00 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a7259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a741090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a7259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a741090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50200/0 (socket says 172.21.15.102:50200) 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a7259c700 1 -- 172.21.15.102:0/3817286913 learned_addr learned my addr 172.21.15.102:0/3817286913 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:20.163 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a7259c700 1 -- 172.21.15.102:0/3817286913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a74109630 con 0x7f4a74106d00 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a7259c700 1 --2- 172.21.15.102:0/3817286913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a741090f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4a640098a0 tx=0x7f4a64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9fee741142a75f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a71d9b700 1 -- 172.21.15.102:0/3817286913 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a64004160 con 0x7f4a74106d00 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a71d9b700 1 -- 172.21.15.102:0/3817286913 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f4a64004300 con 0x7f4a74106d00 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.188+0000 7f4a71d9b700 1 -- 172.21.15.102:0/3817286913 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a64011a60 con 0x7f4a74106d00 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.189+0000 7f4a78de0700 1 -- 172.21.15.102:0/3817286913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 msgr2=0x7f4a741090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.189+0000 7f4a78de0700 1 --2- 172.21.15.102:0/3817286913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a741090f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4a640098a0 tx=0x7f4a64009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.189+0000 7f4a78de0700 1 -- 172.21.15.102:0/3817286913 shutdown_connections 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.189+0000 7f4a78de0700 1 --2- 172.21.15.102:0/3817286913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a741090f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.189+0000 7f4a78de0700 1 -- 172.21.15.102:0/3817286913 >> 172.21.15.102:0/3817286913 conn(0x7f4a741006d0 msgr2=0x7f4a74102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.189+0000 7f4a78de0700 1 -- 172.21.15.102:0/3817286913 shutdown_connections 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.189+0000 7f4a78de0700 1 -- 172.21.15.102:0/3817286913 wait complete. 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a78de0700 1 Processor -- start 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a78de0700 1 -- start start 2024-03-30T13:51:20.164 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a78de0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a7419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a78de0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7410a5a0 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a7259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a7419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a7259c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a7419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50216/0 (socket says 172.21.15.102:50216) 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a7259c700 1 -- 172.21.15.102:0/3262052406 learned_addr learned my addr 172.21.15.102:0/3262052406 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.190+0000 7f4a7259c700 1 -- 172.21.15.102:0/3262052406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7419a620 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a7259c700 1 --2- 172.21.15.102:0/3262052406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a7419a0e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009870 tx=0x7f4a64011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a6bfff700 1 -- 172.21.15.102:0/3262052406 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a6401a430 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a6bfff700 1 -- 172.21.15.102:0/3262052406 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f4a6401aa60 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7419a8b0 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a6bfff700 1 -- 172.21.15.102:0/3262052406 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a640199a0 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a7419d5a0 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a6bfff700 1 -- 172.21.15.102:0/3262052406 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4a64022020 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.191+0000 7f4a6bfff700 1 -- 172.21.15.102:0/3262052406 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4a6402c3a0 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.192+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a74063870 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.194+0000 7f4a6bfff700 1 -- 172.21.15.102:0/3262052406 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4a64004300 con 0x7f4a74106d00 2024-03-30T13:51:20.165 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.261+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f4a7419d980 con 0x7f4a74106d00 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.263+0000 7f4a6bfff700 1 -- 172.21.15.102:0/3262052406 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f4a6401ac10 con 0x7f4a74106d00 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.264+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 msgr2=0x7f4a7419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.264+0000 7f4a78de0700 1 --2- 172.21.15.102:0/3262052406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a7419a0e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009870 tx=0x7f4a64011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.264+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 shutdown_connections 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.264+0000 7f4a78de0700 1 --2- 172.21.15.102:0/3262052406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a74106d00 0x7f4a7419a0e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.264+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 >> 172.21.15.102:0/3262052406 conn(0x7f4a741006d0 msgr2=0x7f4a74101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.264+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 shutdown_connections 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:19.264+0000 7f4a78de0700 1 -- 172.21.15.102:0/3262052406 wait complete. 2024-03-30T13:51:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-30T13:51:20.180 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-30T13:51:20.180 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mgr... 2024-03-30T13:51:20.181 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-30T13:51:20.181 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-30T13:51:20.417 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T13:51:20.417 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.y.service: Unit ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.y.service not loaded. 2024-03-30T13:51:20.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3262052406' entity='client.admin' 2024-03-30T13:51:20.568 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4.target.wants/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.y.service → /etc/systemd/system/ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@.service. 2024-03-30T13:51:20.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:20 smithi102 systemd[1]: Starting Ceph mgr.y for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:51:21.056 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:20 smithi102 podman[29607]: 2024-03-30 13:51:20.700777312 +0000 UTC m=+0.017775933 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T13:51:21.056 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:20 smithi102 podman[29607]: 2024-03-30 13:51:20.816724351 +0000 UTC m=+0.133723015 container create 358299fce19e985026c5c10793060d21f81f8dd64afff573b67fee8a3f72f6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc) 2024-03-30T13:51:21.056 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:20 smithi102 podman[29607]: 2024-03-30 13:51:20.985689625 +0000 UTC m=+0.302688229 container init 358299fce19e985026c5c10793060d21f81f8dd64afff573b67fee8a3f72f6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-30T13:51:21.056 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:20 smithi102 podman[29607]: 2024-03-30 13:51:20.988267711 +0000 UTC m=+0.305266320 container start 358299fce19e985026c5c10793060d21f81f8dd64afff573b67fee8a3f72f6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc) 2024-03-30T13:51:21.057 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:21 smithi102 bash[29607]: 358299fce19e985026c5c10793060d21f81f8dd64afff573b67fee8a3f72f6de 2024-03-30T13:51:21.057 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:21 smithi102 systemd[1]: Started Ceph mgr.y for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:51:21.061 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:51:21.061 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-30T13:51:21.069 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:51:21.069 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-30T13:51:21.069 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-30T13:51:21.069 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T13:51:21.078 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:51:21.078 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-30T13:51:21.086 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:51:21.086 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-30T13:51:21.086 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-30T13:51:21.086 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-30T13:51:21.086 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr to start... 2024-03-30T13:51:21.086 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr... 2024-03-30T13:51:21.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3160162141' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:51:21.943 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:21 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:21.566+0000 7fe676d64200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:51:21.943 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:21 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:21.665+0000 7fe676d64200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:51:22.286 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:22.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:51:22.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "69dd7ec8-ee9c-11ee-b647-cb9ed24678a4", 2024-03-30T13:51:22.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:51:22.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:51:22.296 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:51:22.297 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:51:22.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:51:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:51:13.147920+0000", 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.616+0000 7ffbea24a700 1 Processor -- start 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.616+0000 7ffbea24a700 1 -- start start 2024-03-30T13:51:22.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.616+0000 7ffbea24a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe4109180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.616+0000 7ffbea24a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4079f30 con 0x7ffbe4106d90 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.616+0000 7ffbe37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe4109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.616+0000 7ffbe37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe4109180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44640/0 (socket says 172.21.15.102:44640) 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.616+0000 7ffbe37fe700 1 -- 172.21.15.102:0/3263207958 learned_addr learned my addr 172.21.15.102:0/3263207958 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.617+0000 7ffbe37fe700 1 -- 172.21.15.102:0/3263207958 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe41096c0 con 0x7ffbe4106d90 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.617+0000 7ffbe37fe700 1 --2- 172.21.15.102:0/3263207958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe4109180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40098a0 tx=0x7ffbd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a62c5778bc081f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.617+0000 7ffbe27fc700 1 -- 172.21.15.102:0/3263207958 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbd4004160 con 0x7ffbe4106d90 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.617+0000 7ffbe27fc700 1 -- 172.21.15.102:0/3263207958 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7ffbd4004300 con 0x7ffbe4106d90 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.617+0000 7ffbe27fc700 1 -- 172.21.15.102:0/3263207958 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbd4011a60 con 0x7ffbe4106d90 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.618+0000 7ffbea24a700 1 -- 172.21.15.102:0/3263207958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 msgr2=0x7ffbe4109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.618+0000 7ffbea24a700 1 --2- 172.21.15.102:0/3263207958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe4109180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40098a0 tx=0x7ffbd4009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.618+0000 7ffbea24a700 1 -- 172.21.15.102:0/3263207958 shutdown_connections 2024-03-30T13:51:22.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.618+0000 7ffbea24a700 1 --2- 172.21.15.102:0/3263207958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe4109180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.618+0000 7ffbea24a700 1 -- 172.21.15.102:0/3263207958 >> 172.21.15.102:0/3263207958 conn(0x7ffbe4100720 msgr2=0x7ffbe4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.618+0000 7ffbea24a700 1 -- 172.21.15.102:0/3263207958 shutdown_connections 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.618+0000 7ffbea24a700 1 -- 172.21.15.102:0/3263207958 wait complete. 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.619+0000 7ffbea24a700 1 Processor -- start 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.619+0000 7ffbea24a700 1 -- start start 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.619+0000 7ffbea24a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.619+0000 7ffbea24a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe410a5d0 con 0x7ffbe4106d90 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.619+0000 7ffbe37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.619+0000 7ffbe37fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44652/0 (socket says 172.21.15.102:44652) 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.619+0000 7ffbe37fe700 1 -- 172.21.15.102:0/3160162141 learned_addr learned my addr 172.21.15.102:0/3160162141 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbe37fe700 1 -- 172.21.15.102:0/3160162141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe419a690 con 0x7ffbe4106d90 2024-03-30T13:51:22.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbe37fe700 1 --2- 172.21.15.102:0/3160162141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe419a150 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4011700 tx=0x7ffbd4011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbe0ff9700 1 -- 172.21.15.102:0/3160162141 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbd4003aa0 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbe0ff9700 1 -- 172.21.15.102:0/3160162141 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7ffbd401a8c0 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe419a920 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbe0ff9700 1 -- 172.21.15.102:0/3160162141 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbd40248a0 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe419d610 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.620+0000 7ffbe0ff9700 1 -- 172.21.15.102:0/3160162141 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffbd4022020 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.621+0000 7ffbe0ff9700 1 -- 172.21.15.102:0/3160162141 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffbd402d450 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.621+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbe4063870 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.623+0000 7ffbe0ff9700 1 -- 172.21.15.102:0/3160162141 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ffbd4043dd0 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.696+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ffbe419da70 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.697+0000 7ffbe0ff9700 1 -- 172.21.15.102:0/3160162141 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ffbd4032070 con 0x7ffbe4106d90 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.698+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 msgr2=0x7ffbe419a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.698+0000 7ffbea24a700 1 --2- 172.21.15.102:0/3160162141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe419a150 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4011700 tx=0x7ffbd4011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.698+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 shutdown_connections 2024-03-30T13:51:22.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.698+0000 7ffbea24a700 1 --2- 172.21.15.102:0/3160162141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbe4106d90 0x7ffbe419a150 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:22.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.698+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 >> 172.21.15.102:0/3160162141 conn(0x7ffbe4100720 msgr2=0x7ffbe4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:22.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.698+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 shutdown_connections 2024-03-30T13:51:22.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:21.698+0000 7ffbea24a700 1 -- 172.21.15.102:0/3160162141 wait complete. 2024-03-30T13:51:22.304 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (1/15)... 2024-03-30T13:51:22.571 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:22 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:22.340+0000 7fe676d64200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:51:22.571 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:22 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:22.432+0000 7fe676d64200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:51:22.942 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:22 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:22.571+0000 7fe676d64200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:51:22.943 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:22 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:22.665+0000 7fe676d64200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:51:22.943 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:22 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:22.748+0000 7fe676d64200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:51:23.296 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:23.008+0000 7fe676d64200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:51:23.297 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:23.117+0000 7fe676d64200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:51:23.297 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:23.203+0000 7fe676d64200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:51:23.297 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:23.296+0000 7fe676d64200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:51:23.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:23.471+0000 7fe676d64200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:51:24.678 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:24.443+0000 7fe676d64200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:51:24.678 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:24.574+0000 7fe676d64200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:51:24.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1273040233' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:51:24.943 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:24.915+0000 7fe676d64200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:51:25.384 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:25.003+0000 7fe676d64200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:51:25.384 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:25.201+0000 7fe676d64200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:51:25.384 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:25.384+0000 7fe676d64200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:51:25.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "69dd7ec8-ee9c-11ee-b647-cb9ed24678a4", 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:51:25.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:51:25.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:51:25.414 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:25.415 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:51:13.147920+0000", 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.711+0000 7fa63cee4700 1 Processor -- start 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.711+0000 7fa63cee4700 1 -- start start 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63cee4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa6381090e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63cee4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa638079740 con 0x7fa638106cf0 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa6381090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa6381090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44658/0 (socket says 172.21.15.102:44658) 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63659c700 1 -- 172.21.15.102:0/4268190344 learned_addr learned my addr 172.21.15.102:0/4268190344 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:25.416 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63659c700 1 -- 172.21.15.102:0/4268190344 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa638109620 con 0x7fa638106cf0 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63659c700 1 --2- 172.21.15.102:0/4268190344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa6381090e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa6280098a0 tx=0x7fa628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca68b7cf18576ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63559a700 1 -- 172.21.15.102:0/4268190344 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa628004160 con 0x7fa638106cf0 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63559a700 1 -- 172.21.15.102:0/4268190344 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fa628004300 con 0x7fa638106cf0 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.712+0000 7fa63559a700 1 -- 172.21.15.102:0/4268190344 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa628011a60 con 0x7fa638106cf0 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.713+0000 7fa63cee4700 1 -- 172.21.15.102:0/4268190344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 msgr2=0x7fa6381090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.713+0000 7fa63cee4700 1 --2- 172.21.15.102:0/4268190344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa6381090e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa6280098a0 tx=0x7fa628009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.713+0000 7fa63cee4700 1 -- 172.21.15.102:0/4268190344 shutdown_connections 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.713+0000 7fa63cee4700 1 --2- 172.21.15.102:0/4268190344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa6381090e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.713+0000 7fa63cee4700 1 -- 172.21.15.102:0/4268190344 >> 172.21.15.102:0/4268190344 conn(0x7fa638100740 msgr2=0x7fa638102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.713+0000 7fa63cee4700 1 -- 172.21.15.102:0/4268190344 shutdown_connections 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.713+0000 7fa63cee4700 1 -- 172.21.15.102:0/4268190344 wait complete. 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.714+0000 7fa63cee4700 1 Processor -- start 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.714+0000 7fa63cee4700 1 -- start start 2024-03-30T13:51:25.417 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.714+0000 7fa63cee4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa63819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.714+0000 7fa63cee4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa63810a570 con 0x7fa638106cf0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.714+0000 7fa63659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa63819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.714+0000 7fa63659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa63819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44660/0 (socket says 172.21.15.102:44660) 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.714+0000 7fa63659c700 1 -- 172.21.15.102:0/1273040233 learned_addr learned my addr 172.21.15.102:0/1273040233 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa63659c700 1 -- 172.21.15.102:0/1273040233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63819a650 con 0x7fa638106cf0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa63659c700 1 --2- 172.21.15.102:0/1273040233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa63819a110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa628009870 tx=0x7fa628011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa6277fe700 1 -- 172.21.15.102:0/1273040233 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa628003af0 con 0x7fa638106cf0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63819a940 con 0x7fa638106cf0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa6277fe700 1 -- 172.21.15.102:0/1273040233 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fa62801a8c0 con 0x7fa638106cf0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa6277fe700 1 -- 172.21.15.102:0/1273040233 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa628019960 con 0x7fa638106cf0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63819d5d0 con 0x7fa638106cf0 2024-03-30T13:51:25.418 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa6277fe700 1 -- 172.21.15.102:0/1273040233 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa628022020 con 0x7fa638106cf0 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.715+0000 7fa6277fe700 1 -- 172.21.15.102:0/1273040233 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa62802c3a0 con 0x7fa638106cf0 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.716+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa638063870 con 0x7fa638106cf0 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.718+0000 7fa6277fe700 1 -- 172.21.15.102:0/1273040233 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa62801aa70 con 0x7fa638106cf0 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.790+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa63819da10 con 0x7fa638106cf0 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.790+0000 7fa6277fe700 1 -- 172.21.15.102:0/1273040233 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa62801a430 con 0x7fa638106cf0 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.791+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 msgr2=0x7fa63819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.791+0000 7fa63cee4700 1 --2- 172.21.15.102:0/1273040233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa63819a110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa628009870 tx=0x7fa628011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.791+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 shutdown_connections 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.791+0000 7fa63cee4700 1 --2- 172.21.15.102:0/1273040233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa638106cf0 0x7fa63819a110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.791+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 >> 172.21.15.102:0/1273040233 conn(0x7fa638100740 msgr2=0x7fa6381011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.791+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 shutdown_connections 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:24.791+0000 7fa63cee4700 1 -- 172.21.15.102:0/1273040233 wait complete. 2024-03-30T13:51:25.419 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (2/15)... 2024-03-30T13:51:25.921 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:25.663+0000 7fe676d64200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:51:25.921 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:25.921+0000 7fe676d64200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:51:26.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:26.152+0000 7fe676d64200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:51:27.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:26.941+0000 7fe676d64200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:51:27.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:27.508+0000 7fe676d64200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:51:27.785 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:27.605+0000 7fe676d64200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:27.828+0000 7fe676d64200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: Activating manager daemon y 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: mgrmap e2: y(active, starting, since 0.00314211s) 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:51:28.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: Manager daemon y is now available 2024-03-30T13:51:28.194 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:51:28.194 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:51:28.194 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' 2024-03-30T13:51:28.194 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' 2024-03-30T13:51:28.194 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:27 smithi102 ceph-mon[29392]: from='mgr.14100 172.21.15.102:0/4273635090' entity='mgr.y' 2024-03-30T13:51:28.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "69dd7ec8-ee9c-11ee-b647-cb9ed24678a4", 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:28.535 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:51:28.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:51:28.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:51:28.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:51:13.147920+0000", 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8ddccb700 1 Processor -- start 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8ddccb700 1 -- start start 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8ddccb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d8109100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:28.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8ddccb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d8079fb0 con 0x7fb8d8106d10 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d8109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d8109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44668/0 (socket says 172.21.15.102:44668) 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8d77fe700 1 -- 172.21.15.102:0/1016003657 learned_addr learned my addr 172.21.15.102:0/1016003657 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.808+0000 7fb8d77fe700 1 -- 172.21.15.102:0/1016003657 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d8109640 con 0x7fb8d8106d10 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8d77fe700 1 --2- 172.21.15.102:0/1016003657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d8109100 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80098a0 tx=0x7fb8c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37564e442900695d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8d67fc700 1 -- 172.21.15.102:0/1016003657 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8c8004160 con 0x7fb8d8106d10 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8d67fc700 1 -- 172.21.15.102:0/1016003657 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fb8c8004300 con 0x7fb8d8106d10 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8d67fc700 1 -- 172.21.15.102:0/1016003657 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8c8011a60 con 0x7fb8d8106d10 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8ddccb700 1 -- 172.21.15.102:0/1016003657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 msgr2=0x7fb8d8109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8ddccb700 1 --2- 172.21.15.102:0/1016003657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d8109100 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80098a0 tx=0x7fb8c8009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8ddccb700 1 -- 172.21.15.102:0/1016003657 shutdown_connections 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8ddccb700 1 --2- 172.21.15.102:0/1016003657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d8109100 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8ddccb700 1 -- 172.21.15.102:0/1016003657 >> 172.21.15.102:0/1016003657 conn(0x7fb8d8100720 msgr2=0x7fb8d8102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8ddccb700 1 -- 172.21.15.102:0/1016003657 shutdown_connections 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.809+0000 7fb8ddccb700 1 -- 172.21.15.102:0/1016003657 wait complete. 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.810+0000 7fb8ddccb700 1 Processor -- start 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.810+0000 7fb8ddccb700 1 -- start start 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.810+0000 7fb8ddccb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.810+0000 7fb8ddccb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d810a600 con 0x7fb8d8106d10 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44684/0 (socket says 172.21.15.102:44684) 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d77fe700 1 -- 172.21.15.102:0/43685220 learned_addr learned my addr 172.21.15.102:0/43685220 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d77fe700 1 -- 172.21.15.102:0/43685220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d819a690 con 0x7fb8d8106d10 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d77fe700 1 --2- 172.21.15.102:0/43685220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d819a150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009870 tx=0x7fb8c8011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8c8003af0 con 0x7fb8d8106d10 2024-03-30T13:51:28.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fb8c801a8c0 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8d819a920 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8d819d610 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8c8019960 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb8c8022020 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.811+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb8c802c3a0 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.812+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8c4005150 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.814+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb8c801adc0 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.831+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 2) ==== 45872+0+0 (secure 0 0 0) 0x7fb8c801a430 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.896+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb8c4005710 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.897+0000 7fb8d4ff9700 1 -- 172.21.15.102:0/43685220 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fb8c8028650 con 0x7fb8d8106d10 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.897+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 msgr2=0x7fb8d819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.897+0000 7fb8ddccb700 1 --2- 172.21.15.102:0/43685220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d819a150 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009870 tx=0x7fb8c8011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.898+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 shutdown_connections 2024-03-30T13:51:28.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.898+0000 7fb8ddccb700 1 --2- 172.21.15.102:0/43685220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8d8106d10 0x7fb8d819a150 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:28.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.898+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 >> 172.21.15.102:0/43685220 conn(0x7fb8d8100720 msgr2=0x7fb8d81011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:28.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.898+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 shutdown_connections 2024-03-30T13:51:28.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:27.898+0000 7fb8ddccb700 1 -- 172.21.15.102:0/43685220 wait complete. 2024-03-30T13:51:28.543 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (3/15)... 2024-03-30T13:51:29.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/43685220' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:51:29.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:28 smithi102 ceph-mon[29392]: mgrmap e3: y(active, since 1.00684s) 2024-03-30T13:51:30.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:30 smithi102 ceph-mon[29392]: mgrmap e4: y(active, since 2s) 2024-03-30T13:51:31.783 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "69dd7ec8-ee9c-11ee-b647-cb9ed24678a4", 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:31.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T13:51:31.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T13:51:31.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T13:51:31.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:31.793 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T13:51:13.147920+0000", 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-03-30T13:51:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f932d099700 1 Processor -- start 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f932d099700 1 -- start start 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f932d099700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f9328106b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f932d099700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9328107120 con 0x7f9328106730 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f9326d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f9328106b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f9326d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f9328106b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60560/0 (socket says 172.21.15.102:60560) 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f9326d9d700 1 -- 172.21.15.102:0/2769444504 learned_addr learned my addr 172.21.15.102:0/2769444504 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f9326d9d700 1 -- 172.21.15.102:0/2769444504 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9328105340 con 0x7f9328106730 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.977+0000 7f9326d9d700 1 --2- 172.21.15.102:0/2769444504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f9328106b50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93200098a0 tx=0x7f9320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b7b7eaca38c2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f9325d9b700 1 -- 172.21.15.102:0/2769444504 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9320004160 con 0x7f9328106730 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f9325d9b700 1 -- 172.21.15.102:0/2769444504 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f9320004300 con 0x7f9328106730 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f9325d9b700 1 -- 172.21.15.102:0/2769444504 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9320011a60 con 0x7f9328106730 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f932d099700 1 -- 172.21.15.102:0/2769444504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 msgr2=0x7f9328106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:31.795 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f932d099700 1 --2- 172.21.15.102:0/2769444504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f9328106b50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93200098a0 tx=0x7f9320009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f932d099700 1 -- 172.21.15.102:0/2769444504 shutdown_connections 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f932d099700 1 --2- 172.21.15.102:0/2769444504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f9328106b50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f932d099700 1 -- 172.21.15.102:0/2769444504 >> 172.21.15.102:0/2769444504 conn(0x7f93280ffe90 msgr2=0x7f93281022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f932d099700 1 -- 172.21.15.102:0/2769444504 shutdown_connections 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.978+0000 7f932d099700 1 -- 172.21.15.102:0/2769444504 wait complete. 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.979+0000 7f932d099700 1 Processor -- start 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.979+0000 7f932d099700 1 -- start start 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.979+0000 7f932d099700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f932819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.979+0000 7f932d099700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9328109700 con 0x7f9328106730 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.979+0000 7f9326d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f932819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.979+0000 7f9326d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f932819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60574/0 (socket says 172.21.15.102:60574) 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.979+0000 7f9326d9d700 1 -- 172.21.15.102:0/456452070 learned_addr learned my addr 172.21.15.102:0/456452070 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.980+0000 7f9326d9d700 1 -- 172.21.15.102:0/456452070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932819a6a0 con 0x7f9328106730 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.980+0000 7f9326d9d700 1 --2- 172.21.15.102:0/456452070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f932819a160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9320011700 tx=0x7f9320011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:31.796 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.980+0000 7f930ffff700 1 -- 172.21.15.102:0/456452070 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93200039e0 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.980+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932819a930 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.980+0000 7f930ffff700 1 -- 172.21.15.102:0/456452070 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f932001a8c0 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.980+0000 7f930ffff700 1 -- 172.21.15.102:0/456452070 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9320024810 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.980+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932819d620 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.981+0000 7f930ffff700 1 -- 172.21.15.102:0/456452070 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46070+0+0 (secure 0 0 0) 0x7f9320022020 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.981+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9328063870 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.981+0000 7f930ffff700 1 --2- 172.21.15.102:0/456452070 >> v2:172.21.15.102:6800/2520146638 conn(0x7f9310038d20 0x7f931003b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.981+0000 7f930ffff700 1 -- 172.21.15.102:0/456452070 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f932004cdb0 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.981+0000 7f932659c700 1 --2- 172.21.15.102:0/456452070 >> v2:172.21.15.102:6800/2520146638 conn(0x7f9310038d20 0x7f931003b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.982+0000 7f932659c700 1 --2- 172.21.15.102:0/456452070 >> v2:172.21.15.102:6800/2520146638 conn(0x7f9310038d20 0x7f931003b1e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f931c00ad30 tx=0x7f931c0093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:30.986+0000 7f930ffff700 1 -- 172.21.15.102:0/456452070 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9320051070 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.255+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f932802dc00 con 0x7f9328106730 2024-03-30T13:51:31.797 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.256+0000 7f930ffff700 1 -- 172.21.15.102:0/456452070 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f932002d4a0 con 0x7f9328106730 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 >> v2:172.21.15.102:6800/2520146638 conn(0x7f9310038d20 msgr2=0x7f931003b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 --2- 172.21.15.102:0/456452070 >> v2:172.21.15.102:6800/2520146638 conn(0x7f9310038d20 0x7f931003b1e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f931c00ad30 tx=0x7f931c0093f0 comp rx=0 tx=0).stop 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 msgr2=0x7f932819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 --2- 172.21.15.102:0/456452070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f932819a160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9320011700 tx=0x7f9320011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 shutdown_connections 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 --2- 172.21.15.102:0/456452070 >> v2:172.21.15.102:6800/2520146638 conn(0x7f9310038d20 0x7f931003b1e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 --2- 172.21.15.102:0/456452070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9328106730 0x7f932819a160 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 >> 172.21.15.102:0/456452070 conn(0x7f93280ffe90 msgr2=0x7f9328100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 shutdown_connections 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:31.258+0000 7f932d099700 1 -- 172.21.15.102:0/456452070 wait complete. 2024-03-30T13:51:31.798 INFO:teuthology.orchestra.run.smithi102.stdout:mgr is available 2024-03-30T13:51:31.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/456452070' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:32.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.245+0000 7f3b2729f700 1 Processor -- start 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.245+0000 7f3b2729f700 1 -- start start 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.245+0000 7f3b2729f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b201066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.245+0000 7f3b2729f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20106c70 con 0x7f3b20106280 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b2503b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b201066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b2503b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b201066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60576/0 (socket says 172.21.15.102:60576) 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b2503b700 1 -- 172.21.15.102:0/703629853 learned_addr learned my addr 172.21.15.102:0/703629853 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:32.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b2503b700 1 -- 172.21.15.102:0/703629853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b20107480 con 0x7f3b20106280 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b2503b700 1 --2- 172.21.15.102:0/703629853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b201066a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0098a0 tx=0x7f3b1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772674a49ecfa783 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b17fff700 1 -- 172.21.15.102:0/703629853 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1c004160 con 0x7f3b20106280 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b17fff700 1 -- 172.21.15.102:0/703629853 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f3b1c004300 con 0x7f3b20106280 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.246+0000 7f3b17fff700 1 -- 172.21.15.102:0/703629853 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1c011a60 con 0x7f3b20106280 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 -- 172.21.15.102:0/703629853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 msgr2=0x7f3b201066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 --2- 172.21.15.102:0/703629853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b201066a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0098a0 tx=0x7f3b1c009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 -- 172.21.15.102:0/703629853 shutdown_connections 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 --2- 172.21.15.102:0/703629853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b201066a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 -- 172.21.15.102:0/703629853 >> 172.21.15.102:0/703629853 conn(0x7f3b20101860 msgr2=0x7f3b20103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 -- 172.21.15.102:0/703629853 shutdown_connections 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 -- 172.21.15.102:0/703629853 wait complete. 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 Processor -- start 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.247+0000 7f3b2729f700 1 -- start start 2024-03-30T13:51:32.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2729f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b2019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2729f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2010a830 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2503b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b2019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2503b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b2019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60592/0 (socket says 172.21.15.102:60592) 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2503b700 1 -- 172.21.15.102:0/533600153 learned_addr learned my addr 172.21.15.102:0/533600153 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2503b700 1 -- 172.21.15.102:0/533600153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2019a6e0 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2503b700 1 --2- 172.21.15.102:0/533600153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b2019a1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009fb0 tx=0x7f3b1c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b167fc700 1 -- 172.21.15.102:0/533600153 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1c0039e0 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2019a970 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b167fc700 1 -- 172.21.15.102:0/533600153 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f3b1c01a8c0 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b167fc700 1 -- 172.21.15.102:0/533600153 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1c024800 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.248+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2019d660 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.249+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b20063870 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.249+0000 7f3b167fc700 1 -- 172.21.15.102:0/533600153 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46070+0+0 (secure 0 0 0) 0x7f3b1c022020 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.249+0000 7f3b167fc700 1 --2- 172.21.15.102:0/533600153 >> v2:172.21.15.102:6800/2520146638 conn(0x7f3b0c038d60 0x7f3b0c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.250+0000 7f3b167fc700 1 -- 172.21.15.102:0/533600153 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3b1c04ced0 con 0x7f3b20106280 2024-03-30T13:51:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.250+0000 7f3b2483a700 1 --2- 172.21.15.102:0/533600153 >> v2:172.21.15.102:6800/2520146638 conn(0x7f3b0c038d60 0x7f3b0c03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.250+0000 7f3b2483a700 1 --2- 172.21.15.102:0/533600153 >> v2:172.21.15.102:6800/2520146638 conn(0x7f3b0c038d60 0x7f3b0c03b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b10006fd0 tx=0x7f3b10006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.254+0000 7f3b167fc700 1 -- 172.21.15.102:0/533600153 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3b1c04cc30 con 0x7f3b20106280 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.448+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f3b2019ecb0 con 0x7f3b20106280 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.449+0000 7f3b167fc700 1 -- 172.21.15.102:0/533600153 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+282 (secure 0 0 0) 0x7f3b1c02d580 con 0x7f3b20106280 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 >> v2:172.21.15.102:6800/2520146638 conn(0x7f3b0c038d60 msgr2=0x7f3b0c03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 --2- 172.21.15.102:0/533600153 >> v2:172.21.15.102:6800/2520146638 conn(0x7f3b0c038d60 0x7f3b0c03b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b10006fd0 tx=0x7f3b10006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 msgr2=0x7f3b2019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 --2- 172.21.15.102:0/533600153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b2019a1a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009fb0 tx=0x7f3b1c011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 shutdown_connections 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 --2- 172.21.15.102:0/533600153 >> v2:172.21.15.102:6800/2520146638 conn(0x7f3b0c038d60 0x7f3b0c03b220 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 --2- 172.21.15.102:0/533600153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20106280 0x7f3b2019a1a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 >> 172.21.15.102:0/533600153 conn(0x7f3b20101860 msgr2=0x7f3b20102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 shutdown_connections 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:32.451+0000 7f3b2729f700 1 -- 172.21.15.102:0/533600153 wait complete. 2024-03-30T13:51:32.989 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling cephadm module... 2024-03-30T13:51:33.094 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/533600153' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T13:51:33.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3322148569' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-30T13:51:34.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:33 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ignoring --setuser ceph since I am not root 2024-03-30T13:51:34.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:33 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ignoring --setgroup ceph since I am not root 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.386+0000 7ff1f1cfb700 1 Processor -- start 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.386+0000 7ff1f1cfb700 1 -- start start 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.386+0000 7ff1f1cfb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec106660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.386+0000 7ff1f1cfb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec106c30 con 0x7ff1ec106240 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.386+0000 7ff1eb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.387+0000 7ff1eb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60596/0 (socket says 172.21.15.102:60596) 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.387+0000 7ff1eb7fe700 1 -- 172.21.15.102:0/4232454781 learned_addr learned my addr 172.21.15.102:0/4232454781 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:34.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.387+0000 7ff1eb7fe700 1 -- 172.21.15.102:0/4232454781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec107440 con 0x7ff1ec106240 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.387+0000 7ff1eb7fe700 1 --2- 172.21.15.102:0/4232454781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec106660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0098a0 tx=0x7ff1dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9b35f1a7792e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.387+0000 7ff1eaffd700 1 -- 172.21.15.102:0/4232454781 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1dc004160 con 0x7ff1ec106240 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.387+0000 7ff1eaffd700 1 -- 172.21.15.102:0/4232454781 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7ff1dc004300 con 0x7ff1ec106240 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.387+0000 7ff1eaffd700 1 -- 172.21.15.102:0/4232454781 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1dc011a60 con 0x7ff1ec106240 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/4232454781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 msgr2=0x7ff1ec106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 --2- 172.21.15.102:0/4232454781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec106660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0098a0 tx=0x7ff1dc009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/4232454781 shutdown_connections 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 --2- 172.21.15.102:0/4232454781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec106660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/4232454781 >> 172.21.15.102:0/4232454781 conn(0x7ff1ec101800 msgr2=0x7ff1ec103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/4232454781 shutdown_connections 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/4232454781 wait complete. 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 Processor -- start 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.388+0000 7ff1f1cfb700 1 -- start start 2024-03-30T13:51:34.475 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1f1cfb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec07bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1f1cfb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec10a7f0 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1eb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec07bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1eb7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec07bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60606/0 (socket says 172.21.15.102:60606) 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1eb7fe700 1 -- 172.21.15.102:0/3322148569 learned_addr learned my addr 172.21.15.102:0/3322148569 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1eb7fe700 1 -- 172.21.15.102:0/3322148569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec07c1f0 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1eb7fe700 1 --2- 172.21.15.102:0/3322148569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec07bcb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc011700 tx=0x7ff1dc0046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1dc003b20 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7ff1dc01a8c0 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec07a3e0 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1dc0249a0 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.389+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ec07a920 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.390+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46070+0+0 (secure 0 0 0) 0x7ff1dc022020 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.390+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ec063870 con 0x7ff1ec106240 2024-03-30T13:51:34.476 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.390+0000 7ff1e97fa700 1 --2- 172.21.15.102:0/3322148569 >> v2:172.21.15.102:6800/2520146638 conn(0x7ff1d8038d70 0x7ff1d803b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.390+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff1dc04cd10 con 0x7ff1ec106240 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.390+0000 7ff1e2ffd700 1 --2- 172.21.15.102:0/3322148569 >> v2:172.21.15.102:6800/2520146638 conn(0x7ff1d8038d70 0x7ff1d803b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.391+0000 7ff1e2ffd700 1 --2- 172.21.15.102:0/3322148569 >> v2:172.21.15.102:6800/2520146638 conn(0x7ff1d8038d70 0x7ff1d803b230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4006fd0 tx=0x7ff1d4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.395+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1dc00b3e0 con 0x7ff1ec106240 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.643+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7ff1ec07b0a0 con 0x7ff1ec106240 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.888+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7ff1dc0172b0 con 0x7ff1ec106240 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.890+0000 7ff1e97fa700 1 -- 172.21.15.102:0/3322148569 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 5) ==== 46081+0+0 (secure 0 0 0) 0x7ff1dc02d720 con 0x7ff1ec106240 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 >> v2:172.21.15.102:6800/2520146638 conn(0x7ff1d8038d70 msgr2=0x7ff1d803b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 --2- 172.21.15.102:0/3322148569 >> v2:172.21.15.102:6800/2520146638 conn(0x7ff1d8038d70 0x7ff1d803b230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4006fd0 tx=0x7ff1d4006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 msgr2=0x7ff1ec07bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 --2- 172.21.15.102:0/3322148569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec07bcb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc011700 tx=0x7ff1dc0046a0 comp rx=0 tx=0).stop 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 shutdown_connections 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 --2- 172.21.15.102:0/3322148569 >> v2:172.21.15.102:6800/2520146638 conn(0x7ff1d8038d70 0x7ff1d803b230 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:34.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 --2- 172.21.15.102:0/3322148569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1ec106240 0x7ff1ec07bcb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:34.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 >> 172.21.15.102:0/3322148569 conn(0x7ff1ec101800 msgr2=0x7ff1ec102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:34.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 shutdown_connections 2024-03-30T13:51:34.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:33.892+0000 7ff1f1cfb700 1 -- 172.21.15.102:0/3322148569 wait complete. 2024-03-30T13:51:34.706 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:34 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:34.454+0000 7f8c977fa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:51:34.707 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:34 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:34.553+0000 7f8c977fa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:51:35.105 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3322148569' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-30T13:51:35.106 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:34 smithi102 ceph-mon[29392]: mgrmap e5: y(active, since 6s) 2024-03-30T13:51:35.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:35 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:35.207+0000 7f8c977fa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:51:35.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:35 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:35.300+0000 7f8c977fa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:51:35.807 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:35 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:35.444+0000 7f8c977fa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:51:35.807 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:35 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:35.539+0000 7f8c977fa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:51:35.807 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:35 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:35.623+0000 7f8c977fa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:51:35.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.900+0000 7fc384f93700 1 Processor -- start 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.900+0000 7fc384f93700 1 -- start start 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc384f93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc380106680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc384f93700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc380106c50 con 0x7fc380106260 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc37e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc380106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc37e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc380106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60646/0 (socket says 172.21.15.102:60646) 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc37e59c700 1 -- 172.21.15.102:0/3688936716 learned_addr learned my addr 172.21.15.102:0/3688936716 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc37e59c700 1 -- 172.21.15.102:0/3688936716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc380107460 con 0x7fc380106260 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc37e59c700 1 --2- 172.21.15.102:0/3688936716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc380106680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3700098a0 tx=0x7fc370009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743dd7c53f4c2f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.901+0000 7fc37d59a700 1 -- 172.21.15.102:0/3688936716 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc370004160 con 0x7fc380106260 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.902+0000 7fc37d59a700 1 -- 172.21.15.102:0/3688936716 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fc370004300 con 0x7fc380106260 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.902+0000 7fc37d59a700 1 -- 172.21.15.102:0/3688936716 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc370011a60 con 0x7fc380106260 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.902+0000 7fc384f93700 1 -- 172.21.15.102:0/3688936716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 msgr2=0x7fc380106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.902+0000 7fc384f93700 1 --2- 172.21.15.102:0/3688936716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc380106680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3700098a0 tx=0x7fc370009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.902+0000 7fc384f93700 1 -- 172.21.15.102:0/3688936716 shutdown_connections 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.902+0000 7fc384f93700 1 --2- 172.21.15.102:0/3688936716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc380106680 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.902+0000 7fc384f93700 1 -- 172.21.15.102:0/3688936716 >> 172.21.15.102:0/3688936716 conn(0x7fc380101800 msgr2=0x7fc380103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.903+0000 7fc384f93700 1 -- 172.21.15.102:0/3688936716 shutdown_connections 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.903+0000 7fc384f93700 1 -- 172.21.15.102:0/3688936716 wait complete. 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.903+0000 7fc384f93700 1 Processor -- start 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.903+0000 7fc384f93700 1 -- start start 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.903+0000 7fc384f93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc38007bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:35.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.903+0000 7fc384f93700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc38010a810 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc37e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc38007bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc37e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc38007bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60652/0 (socket says 172.21.15.102:60652) 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc37e59c700 1 -- 172.21.15.102:0/668297453 learned_addr learned my addr 172.21.15.102:0/668297453 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc37e59c700 1 -- 172.21.15.102:0/668297453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38007c1f0 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc37e59c700 1 --2- 172.21.15.102:0/668297453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc38007bcb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc370009fb0 tx=0x7fc370011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc36f7fe700 1 -- 172.21.15.102:0/668297453 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3700039e0 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc36f7fe700 1 -- 172.21.15.102:0/668297453 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fc37001a8c0 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38007a3e0 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc36f7fe700 1 -- 172.21.15.102:0/668297453 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc370024800 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.904+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc38007a920 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.905+0000 7fc36f7fe700 1 -- 172.21.15.102:0/668297453 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 46081+0+0 (secure 0 0 0) 0x7fc370022020 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.905+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc380063870 con 0x7fc380106260 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.905+0000 7fc36f7fe700 1 --2- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 0x7fc36803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:35.818 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.905+0000 7fc36f7fe700 1 -- 172.21.15.102:0/668297453 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc37004cda0 con 0x7fc380106260 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.905+0000 7fc37dd9b700 1 -- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 msgr2=0x7fc36803b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2520146638 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.905+0000 7fc37dd9b700 1 --2- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 0x7fc36803b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:34.910+0000 7fc36f7fe700 1 -- 172.21.15.102:0/668297453 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc37000b3e0 con 0x7fc380106260 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.106+0000 7fc37dd9b700 1 -- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 msgr2=0x7fc36803b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2520146638 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.106+0000 7fc37dd9b700 1 --2- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 0x7fc36803b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.159+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc38007b0a0 con 0x7fc380106260 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.160+0000 7fc36f7fe700 1 -- 172.21.15.102:0/668297453 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc3700172b0 con 0x7fc380106260 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 msgr2=0x7fc36803b2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 --2- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 0x7fc36803b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 msgr2=0x7fc38007bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 --2- 172.21.15.102:0/668297453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc38007bcb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc370009fb0 tx=0x7fc370011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 shutdown_connections 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 --2- 172.21.15.102:0/668297453 >> v2:172.21.15.102:6800/2520146638 conn(0x7fc368038e00 0x7fc36803b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:35.819 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 --2- 172.21.15.102:0/668297453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc380106260 0x7fc38007bcb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:35.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 >> 172.21.15.102:0/668297453 conn(0x7fc380101800 msgr2=0x7fc3801022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:35.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 shutdown_connections 2024-03-30T13:51:35.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:35.162+0000 7fc384f93700 1 -- 172.21.15.102:0/668297453 wait complete. 2024-03-30T13:51:35.820 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-03-30T13:51:35.820 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 5... 2024-03-30T13:51:36.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/668297453' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T13:51:36.095 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:35 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:35.893+0000 7f8c977fa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:51:36.095 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:36.005+0000 7f8c977fa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:51:36.096 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:36.095+0000 7f8c977fa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:51:36.370 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:36.194+0000 7f8c977fa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:51:36.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:36.371+0000 7f8c977fa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:51:37.630 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:37 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:37.315+0000 7f8c977fa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:51:37.630 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:37 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:37.442+0000 7f8c977fa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:51:37.942 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:37 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:37.771+0000 7f8c977fa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:51:37.942 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:37 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:37.858+0000 7f8c977fa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:51:38.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:38 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:38.053+0000 7f8c977fa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:51:38.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:38 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:38.234+0000 7f8c977fa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:51:38.773 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:38 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:38.515+0000 7f8c977fa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:51:38.774 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:38 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:38.771+0000 7f8c977fa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:51:39.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:39 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:39.004+0000 7f8c977fa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:51:40.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:39 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:39.796+0000 7f8c977fa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:51:40.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:40.358+0000 7f8c977fa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:51:40.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:40.455+0000 7f8c977fa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:51:40.669 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:51:40.668+0000 7f8c977fa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:51:40.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: Active manager daemon y restarted 2024-03-30T13:51:40.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: Activating manager daemon y 2024-03-30T13:51:40.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: osdmap e2: 0 total, 0 up, 0 in 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: mgrmap e6: y(active, starting, since 0.00312876s) 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: Manager daemon y is now available 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:51:40.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:40 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:51:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:41 smithi102 ceph-mon[29392]: Found migration_current of "None". Setting to last migration. 2024-03-30T13:51:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:41 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:41 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:41 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:41 smithi102 ceph-mon[29392]: mgrmap e7: y(active, since 1.0059s) 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.223+0000 7fb972881700 1 Processor -- start 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.223+0000 7fb972881700 1 -- start start 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.223+0000 7fb972881700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c109180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.223+0000 7fb972881700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c0796a0 con 0x7fb96c106d90 2024-03-30T13:51:42.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.223+0000 7fb96bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.223+0000 7fb96bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c109180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60666/0 (socket says 172.21.15.102:60666) 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.223+0000 7fb96bfff700 1 -- 172.21.15.102:0/2418235050 learned_addr learned my addr 172.21.15.102:0/2418235050 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.224+0000 7fb96bfff700 1 -- 172.21.15.102:0/2418235050 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c1096c0 con 0x7fb96c106d90 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.224+0000 7fb96bfff700 1 --2- 172.21.15.102:0/2418235050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c109180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0098a0 tx=0x7fb95c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1538ca1b31d6d5df server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.224+0000 7fb96affd700 1 -- 172.21.15.102:0/2418235050 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb95c004160 con 0x7fb96c106d90 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.224+0000 7fb96affd700 1 -- 172.21.15.102:0/2418235050 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fb95c004300 con 0x7fb96c106d90 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.224+0000 7fb96affd700 1 -- 172.21.15.102:0/2418235050 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb95c011a60 con 0x7fb96c106d90 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.224+0000 7fb972881700 1 -- 172.21.15.102:0/2418235050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 msgr2=0x7fb96c109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.224+0000 7fb972881700 1 --2- 172.21.15.102:0/2418235050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c109180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0098a0 tx=0x7fb95c009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 -- 172.21.15.102:0/2418235050 shutdown_connections 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 --2- 172.21.15.102:0/2418235050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c109180 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 -- 172.21.15.102:0/2418235050 >> 172.21.15.102:0/2418235050 conn(0x7fb96c100740 msgr2=0x7fb96c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 -- 172.21.15.102:0/2418235050 shutdown_connections 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 -- 172.21.15.102:0/2418235050 wait complete. 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 Processor -- start 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 -- start start 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.225+0000 7fb972881700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c10a5d0 con 0x7fb96c106d90 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb96bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb96bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60678/0 (socket says 172.21.15.102:60678) 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb96bfff700 1 -- 172.21.15.102:0/1738033749 learned_addr learned my addr 172.21.15.102:0/1738033749 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb96bfff700 1 -- 172.21.15.102:0/1738033749 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c19a690 con 0x7fb96c106d90 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb96bfff700 1 --2- 172.21.15.102:0/1738033749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c19a150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb95c011700 tx=0x7fb95c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb95c003aa0 con 0x7fb96c106d90 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96c19a920 con 0x7fb96c106d90 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fb95c01a8c0 con 0x7fb96c106d90 2024-03-30T13:51:42.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb95c0248a0 con 0x7fb96c106d90 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.226+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96c19d610 con 0x7fb96c106d90 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.227+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 46081+0+0 (secure 0 0 0) 0x7fb95c022020 con 0x7fb96c106d90 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.227+0000 7fb9697fa700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.227+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 --> v2:172.21.15.102:6800/2520146638 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb95803b950 con 0x7fb958038dc0 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.227+0000 7fb96b7fe700 1 -- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 msgr2=0x7fb95803b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2520146638 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.227+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb95c04ce00 con 0x7fb96c106d90 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.227+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.428+0000 7fb96b7fe700 1 -- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 msgr2=0x7fb95803b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2520146638 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.428+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.829+0000 7fb96b7fe700 1 -- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 msgr2=0x7fb95803b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2520146638 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:36.829+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:37.629+0000 7fb96b7fe700 1 -- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 msgr2=0x7fb95803b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2520146638 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:37.629+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:39.230+0000 7fb96b7fe700 1 -- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 msgr2=0x7fb95803b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2520146638 2024-03-30T13:51:42.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:39.230+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:40.672+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 6) ==== 45883+0+0 (secure 0 0 0) 0x7fb95c011d10 con 0x7fb96c106d90 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:40.672+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 msgr2=0x7fb95803b280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:40.672+0000 7fb9697fa700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.673+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 7) ==== 45975+0+0 (secure 0 0 0) 0x7fb95c04e630 con 0x7fb96c106d90 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.673+0000 7fb9697fa700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/3737793096 conn(0x7fb95803c6d0 0x7fb95803eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.673+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 --> v2:172.21.15.102:6800/3737793096 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb95c001d30 con 0x7fb95803c6d0 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.674+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/3737793096 conn(0x7fb95803c6d0 0x7fb95803eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.675+0000 7fb96b7fe700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/3737793096 conn(0x7fb95803c6d0 0x7fb95803eac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb954003a10 tx=0x7fb954009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.675+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fb95c001d30 con 0x7fb95803c6d0 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 --> v2:172.21.15.102:6800/3737793096 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fb96c063870 con 0x7fb95803c6d0 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb9697fa700 1 -- 172.21.15.102:0/1738033749 <== mgr.14120 v2:172.21.15.102:6800/3737793096 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fb96c063870 con 0x7fb95803c6d0 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/3737793096 conn(0x7fb95803c6d0 msgr2=0x7fb95803eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/3737793096 conn(0x7fb95803c6d0 0x7fb95803eac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb954003a10 tx=0x7fb954009290 comp rx=0 tx=0).stop 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 msgr2=0x7fb96c19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 --2- 172.21.15.102:0/1738033749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c19a150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb95c011700 tx=0x7fb95c011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:42.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 shutdown_connections 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/3737793096 conn(0x7fb95803c6d0 0x7fb95803eac0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 --2- 172.21.15.102:0/1738033749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb96c106d90 0x7fb96c19a150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 --2- 172.21.15.102:0/1738033749 >> v2:172.21.15.102:6800/2520146638 conn(0x7fb958038dc0 0x7fb95803b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.682+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 >> 172.21.15.102:0/1738033749 conn(0x7fb96c100740 msgr2=0x7fb96c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.683+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 shutdown_connections 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:41.683+0000 7fb972881700 1 -- 172.21.15.102:0/1738033749 wait complete. 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 5 is available 2024-03-30T13:51:42.413 INFO:teuthology.orchestra.run.smithi102.stdout:Setting orchestrator backend to cephadm... 2024-03-30T13:51:43.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:42 smithi102 ceph-mon[29392]: [30/Mar/2024:13:51:41] ENGINE Bus STARTING 2024-03-30T13:51:43.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:42 smithi102 ceph-mon[29392]: [30/Mar/2024:13:51:41] ENGINE Serving on http://172.21.15.102:8765 2024-03-30T13:51:43.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:42 smithi102 ceph-mon[29392]: [30/Mar/2024:13:51:41] ENGINE Serving on https://172.21.15.102:7150 2024-03-30T13:51:43.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:42 smithi102 ceph-mon[29392]: [30/Mar/2024:13:51:41] ENGINE Bus STARTED 2024-03-30T13:51:43.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:42 smithi102 ceph-mon[29392]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T13:51:43.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:42 smithi102 ceph-mon[29392]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T13:51:43.602 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f7acf7700 1 Processor -- start 2024-03-30T13:51:43.602 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f7acf7700 1 -- start start 2024-03-30T13:51:43.602 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f7acf7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f74106660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:43.602 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f7acf7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4f74106c30 con 0x7f4f74106240 2024-03-30T13:51:43.602 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f78a93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f74106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:43.602 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f78a93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f74106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45488/0 (socket says 172.21.15.102:45488) 2024-03-30T13:51:43.602 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f78a93700 1 -- 172.21.15.102:0/1647212713 learned_addr learned my addr 172.21.15.102:0/1647212713 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f78a93700 1 -- 172.21.15.102:0/1647212713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f74107440 con 0x7f4f74106240 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.821+0000 7f4f78a93700 1 --2- 172.21.15.102:0/1647212713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f74106660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f680098a0 tx=0x7f4f68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c18bf9f8936c38c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f737fe700 1 -- 172.21.15.102:0/1647212713 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f68004160 con 0x7f4f74106240 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f737fe700 1 -- 172.21.15.102:0/1647212713 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f4f68004300 con 0x7f4f74106240 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f737fe700 1 -- 172.21.15.102:0/1647212713 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f68011a60 con 0x7f4f74106240 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f7acf7700 1 -- 172.21.15.102:0/1647212713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 msgr2=0x7f4f74106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f7acf7700 1 --2- 172.21.15.102:0/1647212713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f74106660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f680098a0 tx=0x7f4f68009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f7acf7700 1 -- 172.21.15.102:0/1647212713 shutdown_connections 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f7acf7700 1 --2- 172.21.15.102:0/1647212713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f74106660 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f7acf7700 1 -- 172.21.15.102:0/1647212713 >> 172.21.15.102:0/1647212713 conn(0x7f4f74101800 msgr2=0x7f4f74103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f7acf7700 1 -- 172.21.15.102:0/1647212713 shutdown_connections 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.822+0000 7f4f7acf7700 1 -- 172.21.15.102:0/1647212713 wait complete. 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f7acf7700 1 Processor -- start 2024-03-30T13:51:43.603 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f7acf7700 1 -- start start 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f7acf7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f7419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f7acf7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7410a7f0 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f78a93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f7419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f78a93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f7419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45492/0 (socket says 172.21.15.102:45492) 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f78a93700 1 -- 172.21.15.102:0/850489348 learned_addr learned my addr 172.21.15.102:0/850489348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f78a93700 1 -- 172.21.15.102:0/850489348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7419a640 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.823+0000 7f4f78a93700 1 --2- 172.21.15.102:0/850489348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f7419a100 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f68011700 tx=0x7f4f680046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.824+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f68003b20 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.824+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f4f6801a8c0 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.824+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f680249a0 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.824+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7419a8d0 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.824+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7419d5c0 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.825+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 7) ==== 45975+0+0 (secure 0 0 0) 0x7f4f68022020 con 0x7f4f74106240 2024-03-30T13:51:43.604 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.825+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f74063870 con 0x7f4f74106240 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.825+0000 7f4f71ffb700 1 --2- 172.21.15.102:0/850489348 >> v2:172.21.15.102:6800/3737793096 conn(0x7f4f64038cb0 0x7f4f6403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.825+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f4f6804cb90 con 0x7f4f74106240 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.825+0000 7f4f73fff700 1 --2- 172.21.15.102:0/850489348 >> v2:172.21.15.102:6800/3737793096 conn(0x7f4f64038cb0 0x7f4f6403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.825+0000 7f4f73fff700 1 --2- 172.21.15.102:0/850489348 >> v2:172.21.15.102:6800/3737793096 conn(0x7f4f64038cb0 0x7f4f6403b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4f60006fd0 tx=0x7f4f60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.830+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4f6802c3a0 con 0x7f4f74106240 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:42.925+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f4f6801a430 con 0x7f4f74106240 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.041+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 --> v2:172.21.15.102:6800/3737793096 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f4f74102960 con 0x7f4f64038cb0 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.047+0000 7f4f71ffb700 1 -- 172.21.15.102:0/850489348 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4f74102960 con 0x7f4f64038cb0 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 >> v2:172.21.15.102:6800/3737793096 conn(0x7f4f64038cb0 msgr2=0x7f4f6403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 --2- 172.21.15.102:0/850489348 >> v2:172.21.15.102:6800/3737793096 conn(0x7f4f64038cb0 0x7f4f6403b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4f60006fd0 tx=0x7f4f60006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 msgr2=0x7f4f7419a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 --2- 172.21.15.102:0/850489348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f7419a100 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4f68011700 tx=0x7f4f680046a0 comp rx=0 tx=0).stop 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 shutdown_connections 2024-03-30T13:51:43.605 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 --2- 172.21.15.102:0/850489348 >> v2:172.21.15.102:6800/3737793096 conn(0x7f4f64038cb0 0x7f4f6403b170 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:43.606 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 --2- 172.21.15.102:0/850489348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f74106240 0x7f4f7419a100 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:43.606 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 >> 172.21.15.102:0/850489348 conn(0x7f4f74101800 msgr2=0x7f4f74102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:43.606 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 shutdown_connections 2024-03-30T13:51:43.606 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:43.049+0000 7f4f7acf7700 1 -- 172.21.15.102:0/850489348 wait complete. 2024-03-30T13:51:44.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:43 smithi102 ceph-mon[29392]: mgrmap e8: y(active, since 2s) 2024-03-30T13:51:44.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:43 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:44.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:43 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:44.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-30T13:51:44.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.021+0000 7f1b1c71b700 1 Processor -- start 2024-03-30T13:51:44.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.021+0000 7f1b1c71b700 1 -- start start 2024-03-30T13:51:44.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.021+0000 7f1b1c71b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b141090c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:44.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.021+0000 7f1b1c71b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1b14079760 con 0x7f1b14106cd0 2024-03-30T13:51:44.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b1a4b7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b141090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:44.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b1a4b7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b141090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45508/0 (socket says 172.21.15.102:45508) 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b1a4b7700 1 -- 172.21.15.102:0/4092094349 learned_addr learned my addr 172.21.15.102:0/4092094349 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b1a4b7700 1 -- 172.21.15.102:0/4092094349 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b14109600 con 0x7f1b14106cd0 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b1a4b7700 1 --2- 172.21.15.102:0/4092094349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b141090c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009e80 tx=0x7f1b1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3d6debb901f940 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b194b5700 1 -- 172.21.15.102:0/4092094349 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b10017070 con 0x7f1b14106cd0 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b194b5700 1 -- 172.21.15.102:0/4092094349 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f1b1000b8c0 con 0x7f1b14106cd0 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.022+0000 7f1b194b5700 1 -- 172.21.15.102:0/4092094349 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b10004470 con 0x7f1b14106cd0 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4092094349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 msgr2=0x7f1b141090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 --2- 172.21.15.102:0/4092094349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b141090c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009e80 tx=0x7f1b1000b330 comp rx=0 tx=0).stop 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4092094349 shutdown_connections 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 --2- 172.21.15.102:0/4092094349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b141090c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4092094349 >> 172.21.15.102:0/4092094349 conn(0x7f1b14100740 msgr2=0x7f1b14102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4092094349 shutdown_connections 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4092094349 wait complete. 2024-03-30T13:51:44.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.023+0000 7f1b1c71b700 1 Processor -- start 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1c71b700 1 -- start start 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1c71b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b1419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1c71b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1410a5d0 con 0x7f1b14106cd0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1a4b7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b1419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1a4b7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b1419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45518/0 (socket says 172.21.15.102:45518) 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1a4b7700 1 -- 172.21.15.102:0/4211979682 learned_addr learned my addr 172.21.15.102:0/4211979682 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1a4b7700 1 -- 172.21.15.102:0/4211979682 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1419a6b0 con 0x7f1b14106cd0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1a4b7700 1 --2- 172.21.15.102:0/4211979682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b1419a170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1b100072a0 tx=0x7f1b10009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b0b7fe700 1 -- 172.21.15.102:0/4211979682 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b10021020 con 0x7f1b14106cd0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b1419a940 con 0x7f1b14106cd0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b0b7fe700 1 -- 172.21.15.102:0/4211979682 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f1b10003d40 con 0x7f1b14106cd0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b0b7fe700 1 -- 172.21.15.102:0/4211979682 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b10017040 con 0x7f1b14106cd0 2024-03-30T13:51:44.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.024+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b1419d630 con 0x7f1b14106cd0 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.025+0000 7f1b0b7fe700 1 -- 172.21.15.102:0/4211979682 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f1b10024020 con 0x7f1b14106cd0 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.025+0000 7f1b0b7fe700 1 --2- 172.21.15.102:0/4211979682 >> v2:172.21.15.102:6800/3737793096 conn(0x7f1b00038d70 0x7f1b0003b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.025+0000 7f1b0b7fe700 1 -- 172.21.15.102:0/4211979682 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f1b1000f050 con 0x7f1b14106cd0 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.025+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b14063870 con 0x7f1b14106cd0 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.025+0000 7f1b19cb6700 1 --2- 172.21.15.102:0/4211979682 >> v2:172.21.15.102:6800/3737793096 conn(0x7f1b00038d70 0x7f1b0003b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.026+0000 7f1b19cb6700 1 --2- 172.21.15.102:0/4211979682 >> v2:172.21.15.102:6800/3737793096 conn(0x7f1b00038d70 0x7f1b0003b230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b04006fd0 tx=0x7f1b04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.030+0000 7f1b0b7fe700 1 -- 172.21.15.102:0/4211979682 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b1002b020 con 0x7f1b14106cd0 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.224+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 --> v2:172.21.15.102:6800/3737793096 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f1b14101920 con 0x7f1b00038d70 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.225+0000 7f1b0b7fe700 1 -- 172.21.15.102:0/4211979682 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f1b14101920 con 0x7f1b00038d70 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 >> v2:172.21.15.102:6800/3737793096 conn(0x7f1b00038d70 msgr2=0x7f1b0003b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 --2- 172.21.15.102:0/4211979682 >> v2:172.21.15.102:6800/3737793096 conn(0x7f1b00038d70 0x7f1b0003b230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b04006fd0 tx=0x7f1b04006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 msgr2=0x7f1b1419a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 --2- 172.21.15.102:0/4211979682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b1419a170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1b100072a0 tx=0x7f1b10009610 comp rx=0 tx=0).stop 2024-03-30T13:51:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 shutdown_connections 2024-03-30T13:51:44.788 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 --2- 172.21.15.102:0/4211979682 >> v2:172.21.15.102:6800/3737793096 conn(0x7f1b00038d70 0x7f1b0003b230 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:44.788 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 --2- 172.21.15.102:0/4211979682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b14106cd0 0x7f1b1419a170 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:44.788 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 >> 172.21.15.102:0/4211979682 conn(0x7f1b14100740 msgr2=0x7f1b14101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:44.788 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 shutdown_connections 2024-03-30T13:51:44.788 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:44.227+0000 7f1b1c71b700 1 -- 172.21.15.102:0/4211979682 wait complete. 2024-03-30T13:51:44.788 INFO:teuthology.orchestra.run.smithi102.stdout:Generating ssh key... 2024-03-30T13:51:45.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:44 smithi102 ceph-mon[29392]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:51:45.640 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: Generating public/private rsa key pair. 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: Your identification has been saved in /tmp/tmp3pf1mgy5/key. 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: Your public key has been saved in /tmp/tmp3pf1mgy5/key.pub. 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: The key fingerprint is: 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: SHA256:RIqFrue8c5SD4lRsbl9Jv4Y8zuENEMxd48QskQPdbfY ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: The key's randomart image is: 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: +---[RSA 3072]----+ 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | .ooo*+. | 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | .= +=++.+ | 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | o. = oo.o . | 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | = o. E | 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | = ..oSo | 2024-03-30T13:51:45.641 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | + = +.o . | 2024-03-30T13:51:45.642 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | o * o +o. . | 2024-03-30T13:51:45.642 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | . + oo++o | 2024-03-30T13:51:45.642 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: | .+ .+o. | 2024-03-30T13:51:45.642 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: +----[SHA256]-----+ 2024-03-30T13:51:46.167 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e6cd3700 1 Processor -- start 2024-03-30T13:51:46.167 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e6cd3700 1 -- start start 2024-03-30T13:51:46.167 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e6cd3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e0109150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:46.167 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e6cd3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e0079f70 con 0x7fa2e0106d60 2024-03-30T13:51:46.167 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e4a6f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e0109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e4a6f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e0109150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45524/0 (socket says 172.21.15.102:45524) 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e4a6f700 1 -- 172.21.15.102:0/4169789161 learned_addr learned my addr 172.21.15.102:0/4169789161 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e4a6f700 1 -- 172.21.15.102:0/4169789161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e0109690 con 0x7fa2e0106d60 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.184+0000 7fa2e4a6f700 1 --2- 172.21.15.102:0/4169789161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e0109150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2d40098a0 tx=0x7fa2d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ad5a6bad64e55a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2df7fe700 1 -- 172.21.15.102:0/4169789161 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2d4004160 con 0x7fa2e0106d60 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2df7fe700 1 -- 172.21.15.102:0/4169789161 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fa2d4004300 con 0x7fa2e0106d60 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2df7fe700 1 -- 172.21.15.102:0/4169789161 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2d4011a60 con 0x7fa2e0106d60 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/4169789161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 msgr2=0x7fa2e0109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2e6cd3700 1 --2- 172.21.15.102:0/4169789161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e0109150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2d40098a0 tx=0x7fa2d4009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/4169789161 shutdown_connections 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2e6cd3700 1 --2- 172.21.15.102:0/4169789161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e0109150 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/4169789161 >> 172.21.15.102:0/4169789161 conn(0x7fa2e0100770 msgr2=0x7fa2e0102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/4169789161 shutdown_connections 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.185+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/4169789161 wait complete. 2024-03-30T13:51:46.168 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e6cd3700 1 Processor -- start 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e6cd3700 1 -- start start 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e6cd3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e01065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e6cd3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e010a5f0 con 0x7fa2e0106d60 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e4a6f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e01065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e4a6f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e01065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45538/0 (socket says 172.21.15.102:45538) 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e4a6f700 1 -- 172.21.15.102:0/844126697 learned_addr learned my addr 172.21.15.102:0/844126697 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e4a6f700 1 -- 172.21.15.102:0/844126697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e0106b20 con 0x7fa2e0106d60 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e4a6f700 1 --2- 172.21.15.102:0/844126697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e01065e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4009870 tx=0x7fa2d4011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2ddffb700 1 -- 172.21.15.102:0/844126697 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2d4003af0 con 0x7fa2e0106d60 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2ddffb700 1 -- 172.21.15.102:0/844126697 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fa2d401a8c0 con 0x7fa2e0106d60 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e0104d10 con 0x7fa2e0106d60 2024-03-30T13:51:46.169 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.186+0000 7fa2ddffb700 1 -- 172.21.15.102:0/844126697 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2d4019960 con 0x7fa2e0106d60 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.187+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e0105250 con 0x7fa2e0106d60 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.187+0000 7fa2ddffb700 1 -- 172.21.15.102:0/844126697 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7fa2d4022020 con 0x7fa2e0106d60 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.187+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e0063870 con 0x7fa2e0106d60 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.188+0000 7fa2ddffb700 1 --2- 172.21.15.102:0/844126697 >> v2:172.21.15.102:6800/3737793096 conn(0x7fa2d0038d70 0x7fa2d003b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.188+0000 7fa2ddffb700 1 -- 172.21.15.102:0/844126697 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7fa2d404cdd0 con 0x7fa2e0106d60 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.188+0000 7fa2dffff700 1 --2- 172.21.15.102:0/844126697 >> v2:172.21.15.102:6800/3737793096 conn(0x7fa2d0038d70 0x7fa2d003b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.188+0000 7fa2dffff700 1 --2- 172.21.15.102:0/844126697 >> v2:172.21.15.102:6800/3737793096 conn(0x7fa2d0038d70 0x7fa2d003b230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc006fd0 tx=0x7fa2cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.193+0000 7fa2ddffb700 1 -- 172.21.15.102:0/844126697 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa2d4020e80 con 0x7fa2e0106d60 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.387+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 --> v2:172.21.15.102:6800/3737793096 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fa2e0105a40 con 0x7fa2d0038d70 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.640+0000 7fa2ddffb700 1 -- 172.21.15.102:0/844126697 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa2e0105a40 con 0x7fa2d0038d70 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 >> v2:172.21.15.102:6800/3737793096 conn(0x7fa2d0038d70 msgr2=0x7fa2d003b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 --2- 172.21.15.102:0/844126697 >> v2:172.21.15.102:6800/3737793096 conn(0x7fa2d0038d70 0x7fa2d003b230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc006fd0 tx=0x7fa2cc006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:46.170 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 msgr2=0x7fa2e01065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:46.171 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 --2- 172.21.15.102:0/844126697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e01065e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4009870 tx=0x7fa2d4011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:46.171 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 shutdown_connections 2024-03-30T13:51:46.171 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 --2- 172.21.15.102:0/844126697 >> v2:172.21.15.102:6800/3737793096 conn(0x7fa2d0038d70 0x7fa2d003b230 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:46.171 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 --2- 172.21.15.102:0/844126697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2e0106d60 0x7fa2e01065e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:46.171 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 >> 172.21.15.102:0/844126697 conn(0x7fa2e0100770 msgr2=0x7fa2e0101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:46.171 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 shutdown_connections 2024-03-30T13:51:46.171 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:45.642+0000 7fa2e6cd3700 1 -- 172.21.15.102:0/844126697 wait complete. 2024-03-30T13:51:46.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-mon[29392]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:51:46.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:46.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:45 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:47.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:46 smithi102 ceph-mon[29392]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:51:47.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:46 smithi102 ceph-mon[29392]: Generating ssh key... 2024-03-30T13:51:47.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3pZrsdGebtcUxK8YVy9J+olJ0RyIk4DwBcY1cfbVlg9GiAH1nJ8LexQpPrAauKowJhW0zkmql4PStok7hzOIJkowepHEO3IYKW7Sif7RufWX3RgksrW78efYKQDj1weBU/adup8yIO6XjvtYhM6oqNJCIAtYxljywAS07JXH0kfgGjQm6/V1lVEma/0aW+rGcjQo48MR7vP9ojtGlXJcgR7FJEKlMf6UQkSf3ga9kIP7ejtrRqKLKowqaOMpvl5k1dbIq/rMe/u0Z7larV77IpcuRUwbmuV0X+cL63/fSBFRi+Sx0wdUDd4Vp1nlTHvSyBQS9Ay2y4MinmzDu7XRMKrC5JRyGhiFoHdjhkq0iLOtliZj82VP1d6Utx825lj4wzlJOGT3r3uDbJpwzRdCC94tMlyIjarBEhJL26PUVi7p5htEbIKLAqdenlvmz4wBkNNj62TND2tH4t8Mx/NnkDVyPI7v9U3lid7DZHRuhAJ5T7PMLupOmd5ynSSQKfVU= ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:51:47.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.536+0000 7f8703e68700 1 Processor -- start 2024-03-30T13:51:47.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.536+0000 7f8703e68700 1 -- start start 2024-03-30T13:51:47.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.536+0000 7f8703e68700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc109110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:47.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.536+0000 7f8703e68700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc079fb0 con 0x7f86fc106d20 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8701c04700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc109110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8701c04700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc109110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45550/0 (socket says 172.21.15.102:45550) 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8701c04700 1 -- 172.21.15.102:0/1185922515 learned_addr learned my addr 172.21.15.102:0/1185922515 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8701c04700 1 -- 172.21.15.102:0/1185922515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc109650 con 0x7f86fc106d20 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8701c04700 1 --2- 172.21.15.102:0/1185922515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc109110 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86f80098a0 tx=0x7f86f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3848dcdd8e271d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8700c02700 1 -- 172.21.15.102:0/1185922515 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f8004160 con 0x7f86fc106d20 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8700c02700 1 -- 172.21.15.102:0/1185922515 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f86f8004300 con 0x7f86fc106d20 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.537+0000 7f8700c02700 1 -- 172.21.15.102:0/1185922515 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f8011a60 con 0x7f86fc106d20 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.538+0000 7f8703e68700 1 -- 172.21.15.102:0/1185922515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 msgr2=0x7f86fc109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.538+0000 7f8703e68700 1 --2- 172.21.15.102:0/1185922515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc109110 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86f80098a0 tx=0x7f86f8009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.538+0000 7f8703e68700 1 -- 172.21.15.102:0/1185922515 shutdown_connections 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.538+0000 7f8703e68700 1 --2- 172.21.15.102:0/1185922515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc109110 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.538+0000 7f8703e68700 1 -- 172.21.15.102:0/1185922515 >> 172.21.15.102:0/1185922515 conn(0x7f86fc100770 msgr2=0x7f86fc102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.538+0000 7f8703e68700 1 -- 172.21.15.102:0/1185922515 shutdown_connections 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.538+0000 7f8703e68700 1 -- 172.21.15.102:0/1185922515 wait complete. 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8703e68700 1 Processor -- start 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8703e68700 1 -- start start 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8703e68700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8703e68700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc10a610 con 0x7f86fc106d20 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8701c04700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8701c04700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45562/0 (socket says 172.21.15.102:45562) 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8701c04700 1 -- 172.21.15.102:0/1001682188 learned_addr learned my addr 172.21.15.102:0/1001682188 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8701c04700 1 -- 172.21.15.102:0/1001682188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc19a690 con 0x7f86fc106d20 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8701c04700 1 --2- 172.21.15.102:0/1001682188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc19a150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86f8011700 tx=0x7f86f8011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f86f2ffd700 1 -- 172.21.15.102:0/1001682188 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f8003aa0 con 0x7f86fc106d20 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f86f2ffd700 1 -- 172.21.15.102:0/1001682188 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f86f801a8c0 con 0x7f86fc106d20 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86fc19a920 con 0x7f86fc106d20 2024-03-30T13:51:47.349 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.539+0000 7f86f2ffd700 1 -- 172.21.15.102:0/1001682188 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f80248a0 con 0x7f86fc106d20 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.540+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86fc19d610 con 0x7f86fc106d20 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.540+0000 7f86f2ffd700 1 -- 172.21.15.102:0/1001682188 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f86f8022020 con 0x7f86fc106d20 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.540+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86fc063870 con 0x7f86fc106d20 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.540+0000 7f86f2ffd700 1 --2- 172.21.15.102:0/1001682188 >> v2:172.21.15.102:6800/3737793096 conn(0x7f86e8038dc0 0x7f86e803b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.540+0000 7f86f2ffd700 1 -- 172.21.15.102:0/1001682188 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f86f804ce00 con 0x7f86fc106d20 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.541+0000 7f8701403700 1 --2- 172.21.15.102:0/1001682188 >> v2:172.21.15.102:6800/3737793096 conn(0x7f86e8038dc0 0x7f86e803b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.541+0000 7f8701403700 1 --2- 172.21.15.102:0/1001682188 >> v2:172.21.15.102:6800/3737793096 conn(0x7f86e8038dc0 0x7f86e803b280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86ec006fd0 tx=0x7f86ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.545+0000 7f86f2ffd700 1 -- 172.21.15.102:0/1001682188 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f86f800b3e0 con 0x7f86fc106d20 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.746+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 --> v2:172.21.15.102:6800/3737793096 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f86fc101920 con 0x7f86e8038dc0 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.746+0000 7f86f2ffd700 1 -- 172.21.15.102:0/1001682188 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f86fc101920 con 0x7f86e8038dc0 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 >> v2:172.21.15.102:6800/3737793096 conn(0x7f86e8038dc0 msgr2=0x7f86e803b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 --2- 172.21.15.102:0/1001682188 >> v2:172.21.15.102:6800/3737793096 conn(0x7f86e8038dc0 0x7f86e803b280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86ec006fd0 tx=0x7f86ec006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:47.350 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 msgr2=0x7f86fc19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 --2- 172.21.15.102:0/1001682188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc19a150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86f8011700 tx=0x7f86f8011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 shutdown_connections 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 --2- 172.21.15.102:0/1001682188 >> v2:172.21.15.102:6800/3737793096 conn(0x7f86e8038dc0 0x7f86e803b280 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 --2- 172.21.15.102:0/1001682188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86fc106d20 0x7f86fc19a150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 >> 172.21.15.102:0/1001682188 conn(0x7f86fc100770 msgr2=0x7f86fc101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 shutdown_connections 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:46.748+0000 7f8703e68700 1 -- 172.21.15.102:0/1001682188 wait complete. 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:Adding key to root@localhost authorized_keys... 2024-03-30T13:51:47.351 INFO:teuthology.orchestra.run.smithi102.stdout:Adding host smithi102... 2024-03-30T13:51:48.181 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:47 smithi102 ceph-mon[29392]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:51:49.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:48 smithi102 ceph-mon[29392]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:51:49.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:48 smithi102 ceph-mon[29392]: Deploying cephadm binary to smithi102 2024-03-30T13:51:52.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Added host 'smithi102' with addr '172.21.15.102' 2024-03-30T13:51:52.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1932d35700 1 Processor -- start 2024-03-30T13:51:52.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1932d35700 1 -- start start 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1932d35700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c106660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1932d35700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f192c106c30 con 0x7f192c106240 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1930ad1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1930ad1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45570/0 (socket says 172.21.15.102:45570) 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1930ad1700 1 -- 172.21.15.102:0/3193661643 learned_addr learned my addr 172.21.15.102:0/3193661643 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1930ad1700 1 -- 172.21.15.102:0/3193661643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192c107440 con 0x7f192c106240 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.761+0000 7f1930ad1700 1 --2- 172.21.15.102:0/3193661643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c106660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19200098a0 tx=0x7f1920009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb21204b8b1dcc0e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f192b7fe700 1 -- 172.21.15.102:0/3193661643 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1920004160 con 0x7f192c106240 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f192b7fe700 1 -- 172.21.15.102:0/3193661643 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f1920004300 con 0x7f192c106240 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f192b7fe700 1 -- 172.21.15.102:0/3193661643 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1920011a60 con 0x7f192c106240 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f1932d35700 1 -- 172.21.15.102:0/3193661643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 msgr2=0x7f192c106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:52.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f1932d35700 1 --2- 172.21.15.102:0/3193661643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c106660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19200098a0 tx=0x7f1920009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f1932d35700 1 -- 172.21.15.102:0/3193661643 shutdown_connections 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f1932d35700 1 --2- 172.21.15.102:0/3193661643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c106660 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f1932d35700 1 -- 172.21.15.102:0/3193661643 >> 172.21.15.102:0/3193661643 conn(0x7f192c101800 msgr2=0x7f192c103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f1932d35700 1 -- 172.21.15.102:0/3193661643 shutdown_connections 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.762+0000 7f1932d35700 1 -- 172.21.15.102:0/3193661643 wait complete. 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1932d35700 1 Processor -- start 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1932d35700 1 -- start start 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1932d35700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1932d35700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f192c10a7f0 con 0x7f192c106240 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1930ad1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1930ad1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c19a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45584/0 (socket says 172.21.15.102:45584) 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1930ad1700 1 -- 172.21.15.102:0/3021196721 learned_addr learned my addr 172.21.15.102:0/3021196721 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.763+0000 7f1930ad1700 1 -- 172.21.15.102:0/3021196721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192c19a640 con 0x7f192c106240 2024-03-30T13:51:52.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.764+0000 7f1930ad1700 1 --2- 172.21.15.102:0/3021196721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c19a100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1920009fb0 tx=0x7f19200046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.764+0000 7f1929ffb700 1 -- 172.21.15.102:0/3021196721 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f192001a430 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.764+0000 7f1929ffb700 1 -- 172.21.15.102:0/3021196721 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f192001aa60 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.764+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192c19a8d0 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.764+0000 7f1929ffb700 1 -- 172.21.15.102:0/3021196721 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19200249b0 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.764+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f192c19d5c0 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.765+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f192c063870 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.765+0000 7f1929ffb700 1 -- 172.21.15.102:0/3021196721 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f1920022020 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.765+0000 7f1929ffb700 1 --2- 172.21.15.102:0/3021196721 >> v2:172.21.15.102:6800/3737793096 conn(0x7f191c038e10 0x7f191c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.765+0000 7f1929ffb700 1 -- 172.21.15.102:0/3021196721 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f192004d250 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.765+0000 7f192bfff700 1 --2- 172.21.15.102:0/3021196721 >> v2:172.21.15.102:6800/3737793096 conn(0x7f191c038e10 0x7f191c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.766+0000 7f192bfff700 1 --2- 172.21.15.102:0/3021196721 >> v2:172.21.15.102:6800/3737793096 conn(0x7f191c038e10 0x7f191c03b2d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1918006fd0 tx=0x7f1918006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.770+0000 7f1929ffb700 1 -- 172.21.15.102:0/3021196721 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f192000f2e0 con 0x7f192c106240 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:47.967+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 --> v2:172.21.15.102:6800/3737793096 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}) -- 0x7f192c102960 con 0x7f191c038e10 2024-03-30T13:51:52.733 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.108+0000 7f1929ffb700 1 -- 172.21.15.102:0/3021196721 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f192c102960 con 0x7f191c038e10 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 >> v2:172.21.15.102:6800/3737793096 conn(0x7f191c038e10 msgr2=0x7f191c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 --2- 172.21.15.102:0/3021196721 >> v2:172.21.15.102:6800/3737793096 conn(0x7f191c038e10 0x7f191c03b2d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1918006fd0 tx=0x7f1918006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 msgr2=0x7f192c19a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 --2- 172.21.15.102:0/3021196721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c19a100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1920009fb0 tx=0x7f19200046a0 comp rx=0 tx=0).stop 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 shutdown_connections 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 --2- 172.21.15.102:0/3021196721 >> v2:172.21.15.102:6800/3737793096 conn(0x7f191c038e10 0x7f191c03b2d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 --2- 172.21.15.102:0/3021196721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f192c106240 0x7f192c19a100 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 >> 172.21.15.102:0/3021196721 conn(0x7f192c101800 msgr2=0x7f192c102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 shutdown_connections 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:52.111+0000 7f1932d35700 1 -- 172.21.15.102:0/3021196721 wait complete. 2024-03-30T13:51:52.734 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mon service... 2024-03-30T13:51:53.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:53 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:53.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:53 smithi102 ceph-mon[29392]: Added host smithi102 2024-03-30T13:51:53.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:53 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.187+0000 7fee3028f700 1 Processor -- start 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.187+0000 7fee3028f700 1 -- start start 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.187+0000 7fee3028f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2807a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.187+0000 7fee3028f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee2807a9f0 con 0x7fee2807bfc0 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.187+0000 7fee2e02b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2807a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.187+0000 7fee2e02b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2807a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60366/0 (socket says 172.21.15.102:60366) 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.187+0000 7fee2e02b700 1 -- 172.21.15.102:0/1252408706 learned_addr learned my addr 172.21.15.102:0/1252408706 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.188+0000 7fee2e02b700 1 -- 172.21.15.102:0/1252408706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2807ab70 con 0x7fee2807bfc0 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.188+0000 7fee2e02b700 1 --2- 172.21.15.102:0/1252408706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2807a420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fee240098a0 tx=0x7fee24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2bd6cd739d740 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:54.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.188+0000 7fee2d029700 1 -- 172.21.15.102:0/1252408706 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee24004160 con 0x7fee2807bfc0 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.188+0000 7fee2d029700 1 -- 172.21.15.102:0/1252408706 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fee24004300 con 0x7fee2807bfc0 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.188+0000 7fee2d029700 1 -- 172.21.15.102:0/1252408706 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee24011a60 con 0x7fee2807bfc0 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.188+0000 7fee3028f700 1 -- 172.21.15.102:0/1252408706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 msgr2=0x7fee2807a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.188+0000 7fee3028f700 1 --2- 172.21.15.102:0/1252408706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2807a420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fee240098a0 tx=0x7fee24009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 -- 172.21.15.102:0/1252408706 shutdown_connections 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 --2- 172.21.15.102:0/1252408706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2807a420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 -- 172.21.15.102:0/1252408706 >> 172.21.15.102:0/1252408706 conn(0x7fee28101800 msgr2=0x7fee28103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 -- 172.21.15.102:0/1252408706 shutdown_connections 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 -- 172.21.15.102:0/1252408706 wait complete. 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 Processor -- start 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 -- start start 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee3028f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee2810c810 con 0x7fee2807bfc0 2024-03-30T13:51:54.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.189+0000 7fee2e02b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee2e02b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2819c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60368/0 (socket says 172.21.15.102:60368) 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee2e02b700 1 -- 172.21.15.102:0/450899033 learned_addr learned my addr 172.21.15.102:0/450899033 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee2e02b700 1 -- 172.21.15.102:0/450899033 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2819c800 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee2e02b700 1 --2- 172.21.15.102:0/450899033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2819c2c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee24009870 tx=0x7fee24011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee1f7fe700 1 -- 172.21.15.102:0/450899033 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee2401a430 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2819ca90 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee1f7fe700 1 -- 172.21.15.102:0/450899033 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7fee2401aa60 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee1f7fe700 1 -- 172.21.15.102:0/450899033 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee240199a0 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.190+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2819f780 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.191+0000 7fee1f7fe700 1 -- 172.21.15.102:0/450899033 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7fee24022020 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.191+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee28063870 con 0x7fee2807bfc0 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.191+0000 7fee1f7fe700 1 --2- 172.21.15.102:0/450899033 >> v2:172.21.15.102:6800/3737793096 conn(0x7fee14038d70 0x7fee1403b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:54.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.191+0000 7fee1f7fe700 1 -- 172.21.15.102:0/450899033 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7fee2404cf60 con 0x7fee2807bfc0 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.191+0000 7fee2d82a700 1 --2- 172.21.15.102:0/450899033 >> v2:172.21.15.102:6800/3737793096 conn(0x7fee14038d70 0x7fee1403b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.192+0000 7fee2d82a700 1 --2- 172.21.15.102:0/450899033 >> v2:172.21.15.102:6800/3737793096 conn(0x7fee14038d70 0x7fee1403b230 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee18006fd0 tx=0x7fee18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.196+0000 7fee1f7fe700 1 -- 172.21.15.102:0/450899033 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fee24029250 con 0x7fee2807bfc0 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.417+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 --> v2:172.21.15.102:6800/3737793096 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fee28102920 con 0x7fee14038d70 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.424+0000 7fee1f7fe700 1 -- 172.21.15.102:0/450899033 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fee28102920 con 0x7fee14038d70 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 >> v2:172.21.15.102:6800/3737793096 conn(0x7fee14038d70 msgr2=0x7fee1403b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 --2- 172.21.15.102:0/450899033 >> v2:172.21.15.102:6800/3737793096 conn(0x7fee14038d70 0x7fee1403b230 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee18006fd0 tx=0x7fee18006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 msgr2=0x7fee2819c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 --2- 172.21.15.102:0/450899033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2819c2c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee24009870 tx=0x7fee24011ae0 comp rx=0 tx=0).stop 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 shutdown_connections 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 --2- 172.21.15.102:0/450899033 >> v2:172.21.15.102:6800/3737793096 conn(0x7fee14038d70 0x7fee1403b230 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 --2- 172.21.15.102:0/450899033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee2807bfc0 0x7fee2819c2c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 >> 172.21.15.102:0/450899033 conn(0x7fee28101800 msgr2=0x7fee28102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:54.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 shutdown_connections 2024-03-30T13:51:54.127 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:53.426+0000 7fee3028f700 1 -- 172.21.15.102:0/450899033 wait complete. 2024-03-30T13:51:54.127 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mgr service... 2024-03-30T13:51:54.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:54 smithi102 ceph-mon[29392]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:51:54.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:54 smithi102 ceph-mon[29392]: Saving service mon spec with placement count:5 2024-03-30T13:51:54.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:54 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:56.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:55 smithi102 ceph-mon[29392]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:51:56.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:55 smithi102 ceph-mon[29392]: Saving service mgr spec with placement count:2 2024-03-30T13:51:56.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:55 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:56.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:55 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:56.146 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.564+0000 7f719bfe8700 1 Processor -- start 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.564+0000 7f719bfe8700 1 -- start start 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.564+0000 7f719bfe8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719407a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.564+0000 7f719bfe8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f719407a9f0 con 0x7f719407bfc0 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7199d84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719407a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7199d84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719407a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60372/0 (socket says 172.21.15.102:60372) 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7199d84700 1 -- 172.21.15.102:0/2515003641 learned_addr learned my addr 172.21.15.102:0/2515003641 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7199d84700 1 -- 172.21.15.102:0/2515003641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719407ab70 con 0x7f719407bfc0 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7199d84700 1 --2- 172.21.15.102:0/2515003641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719407a420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f718c009e80 tx=0x7f718c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff707c51e625bddb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7198d82700 1 -- 172.21.15.102:0/2515003641 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c017070 con 0x7f719407bfc0 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7198d82700 1 -- 172.21.15.102:0/2515003641 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f718c00b8c0 con 0x7f719407bfc0 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.565+0000 7f7198d82700 1 -- 172.21.15.102:0/2515003641 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c004470 con 0x7f719407bfc0 2024-03-30T13:51:56.147 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 -- 172.21.15.102:0/2515003641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 msgr2=0x7f719407a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 --2- 172.21.15.102:0/2515003641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719407a420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f718c009e80 tx=0x7f718c00b330 comp rx=0 tx=0).stop 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 -- 172.21.15.102:0/2515003641 shutdown_connections 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 --2- 172.21.15.102:0/2515003641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719407a420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 -- 172.21.15.102:0/2515003641 >> 172.21.15.102:0/2515003641 conn(0x7f7194101800 msgr2=0x7f7194103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 -- 172.21.15.102:0/2515003641 shutdown_connections 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 -- 172.21.15.102:0/2515003641 wait complete. 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 Processor -- start 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.566+0000 7f719bfe8700 1 -- start start 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f719bfe8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719419c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f719bfe8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f719410c810 con 0x7f719407bfc0 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f7199d84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719419c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f7199d84700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719419c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60378/0 (socket says 172.21.15.102:60378) 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f7199d84700 1 -- 172.21.15.102:0/641700962 learned_addr learned my addr 172.21.15.102:0/641700962 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f7199d84700 1 -- 172.21.15.102:0/641700962 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719419c850 con 0x7f719407bfc0 2024-03-30T13:51:56.148 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f7199d84700 1 --2- 172.21.15.102:0/641700962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719419c310 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f718c009fd0 tx=0x7f718c00be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f718affd700 1 -- 172.21.15.102:0/641700962 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c021020 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f718affd700 1 -- 172.21.15.102:0/641700962 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1062+0+0 (secure 0 0 0) 0x7f718c003d00 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719419cae0 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f718affd700 1 -- 172.21.15.102:0/641700962 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f718c017040 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.567+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719419f7d0 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.568+0000 7f718affd700 1 -- 172.21.15.102:0/641700962 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f718c024020 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.568+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7194063870 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.569+0000 7f718affd700 1 --2- 172.21.15.102:0/641700962 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7180038e00 0x7f718003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.569+0000 7f718affd700 1 -- 172.21.15.102:0/641700962 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f718c00f050 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.569+0000 7f7199583700 1 --2- 172.21.15.102:0/641700962 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7180038e00 0x7f718003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.569+0000 7f7199583700 1 --2- 172.21.15.102:0/641700962 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7180038e00 0x7f718003b2c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7184006fd0 tx=0x7f7184006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.574+0000 7f718affd700 1 -- 172.21.15.102:0/641700962 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f718c02b020 con 0x7f719407bfc0 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.784+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 --> v2:172.21.15.102:6800/3737793096 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f71941029b0 con 0x7f7180038e00 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.791+0000 7f718affd700 1 -- 172.21.15.102:0/641700962 <== mgr.14120 v2:172.21.15.102:6800/3737793096 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f71941029b0 con 0x7f7180038e00 2024-03-30T13:51:56.149 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.793+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7180038e00 msgr2=0x7f718003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.793+0000 7f719bfe8700 1 --2- 172.21.15.102:0/641700962 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7180038e00 0x7f718003b2c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7184006fd0 tx=0x7f7184006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 msgr2=0x7f719419c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 --2- 172.21.15.102:0/641700962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719419c310 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f718c009fd0 tx=0x7f718c00be90 comp rx=0 tx=0).stop 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 shutdown_connections 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 --2- 172.21.15.102:0/641700962 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7180038e00 0x7f718003b2c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 --2- 172.21.15.102:0/641700962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f719407bfc0 0x7f719419c310 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 >> 172.21.15.102:0/641700962 conn(0x7f7194101800 msgr2=0x7f71941022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 shutdown_connections 2024-03-30T13:51:56.150 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:54.794+0000 7f719bfe8700 1 -- 172.21.15.102:0/641700962 wait complete. 2024-03-30T13:51:57.451 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.636+0000 7f8525ba0700 1 Processor -- start 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.636+0000 7f8525ba0700 1 -- start start 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.636+0000 7f8525ba0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f8520106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.636+0000 7f8525ba0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8520106a60 con 0x7f8520106070 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.636+0000 7f8524b9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f8520106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.636+0000 7f8524b9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f8520106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60382/0 (socket says 172.21.15.102:60382) 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.636+0000 7f8524b9e700 1 -- 172.21.15.102:0/3729731740 learned_addr learned my addr 172.21.15.102:0/3729731740 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.637+0000 7f8524b9e700 1 -- 172.21.15.102:0/3729731740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8520107270 con 0x7f8520106070 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.637+0000 7f8524b9e700 1 --2- 172.21.15.102:0/3729731740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f8520106490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85140098a0 tx=0x7f8514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c61b74a95240e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.637+0000 7f851f7fe700 1 -- 172.21.15.102:0/3729731740 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8514004160 con 0x7f8520106070 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.637+0000 7f851f7fe700 1 -- 172.21.15.102:0/3729731740 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f8514004300 con 0x7f8520106070 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.637+0000 7f851f7fe700 1 -- 172.21.15.102:0/3729731740 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8514011a00 con 0x7f8520106070 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 -- 172.21.15.102:0/3729731740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 msgr2=0x7f8520106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:57.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 --2- 172.21.15.102:0/3729731740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f8520106490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85140098a0 tx=0x7f8514009bb0 comp rx=0 tx=0).stop 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 -- 172.21.15.102:0/3729731740 shutdown_connections 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 --2- 172.21.15.102:0/3729731740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f8520106490 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 -- 172.21.15.102:0/3729731740 >> 172.21.15.102:0/3729731740 conn(0x7f8520101610 msgr2=0x7f8520103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 -- 172.21.15.102:0/3729731740 shutdown_connections 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 -- 172.21.15.102:0/3729731740 wait complete. 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 Processor -- start 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.638+0000 7f8525ba0700 1 -- start start 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8525ba0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f852019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8525ba0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f852010a620 con 0x7f8520106070 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8524b9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f852019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8524b9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f852019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60384/0 (socket says 172.21.15.102:60384) 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8524b9e700 1 -- 172.21.15.102:0/1749135410 learned_addr learned my addr 172.21.15.102:0/1749135410 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:57.453 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8524b9e700 1 -- 172.21.15.102:0/1749135410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852019a540 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8524b9e700 1 --2- 172.21.15.102:0/1749135410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f852019a000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8514009fb0 tx=0x7f8514011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f851dffb700 1 -- 172.21.15.102:0/1749135410 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8514003aa0 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f851dffb700 1 -- 172.21.15.102:0/1749135410 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f851401a8e0 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f851dffb700 1 -- 172.21.15.102:0/1749135410 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8514024930 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852019a7d0 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.639+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852019d4c0 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.640+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8520063870 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.641+0000 7f851dffb700 1 -- 172.21.15.102:0/1749135410 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f8514022020 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.641+0000 7f851dffb700 1 --2- 172.21.15.102:0/1749135410 >> v2:172.21.15.102:6800/3737793096 conn(0x7f8510038dc0 0x7f851003b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.641+0000 7f851dffb700 1 -- 172.21.15.102:0/1749135410 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f851404ccf0 con 0x7f8520106070 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.641+0000 7f851ffff700 1 --2- 172.21.15.102:0/1749135410 >> v2:172.21.15.102:6800/3737793096 conn(0x7f8510038dc0 0x7f851003b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.641+0000 7f851ffff700 1 --2- 172.21.15.102:0/1749135410 >> v2:172.21.15.102:6800/3737793096 conn(0x7f8510038dc0 0x7f851003b280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f850c006fd0 tx=0x7f850c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:57.454 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.646+0000 7f851dffb700 1 -- 172.21.15.102:0/1749135410 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f851400f2e0 con 0x7f8520106070 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.847+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f852019d8a0 con 0x7f8520106070 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.849+0000 7f851dffb700 1 -- 172.21.15.102:0/1749135410 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f851404ca50 con 0x7f8520106070 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 >> v2:172.21.15.102:6800/3737793096 conn(0x7f8510038dc0 msgr2=0x7f851003b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 --2- 172.21.15.102:0/1749135410 >> v2:172.21.15.102:6800/3737793096 conn(0x7f8510038dc0 0x7f851003b280 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f850c006fd0 tx=0x7f850c006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 msgr2=0x7f852019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 --2- 172.21.15.102:0/1749135410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f852019a000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8514009fb0 tx=0x7f8514011ac0 comp rx=0 tx=0).stop 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 shutdown_connections 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 --2- 172.21.15.102:0/1749135410 >> v2:172.21.15.102:6800/3737793096 conn(0x7f8510038dc0 0x7f851003b280 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 --2- 172.21.15.102:0/1749135410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8520106070 0x7f852019a000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 >> 172.21.15.102:0/1749135410 conn(0x7f8520101610 msgr2=0x7f8520102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 shutdown_connections 2024-03-30T13:51:57.455 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:56.852+0000 7f8525ba0700 1 -- 172.21.15.102:0/1749135410 wait complete. 2024-03-30T13:51:57.806 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:57 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:51:57.806 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1749135410' entity='client.admin' 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.906+0000 7f036758e700 1 Processor -- start 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.906+0000 7f036758e700 1 -- start start 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.906+0000 7f036758e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f0360106bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.906+0000 7f036758e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0360107190 con 0x7f03601067a0 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.906+0000 7f036532a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f0360106bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.906+0000 7f036532a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f0360106bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60390/0 (socket says 172.21.15.102:60390) 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.906+0000 7f036532a700 1 -- 172.21.15.102:0/375837738 learned_addr learned my addr 172.21.15.102:0/375837738 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.907+0000 7f036532a700 1 -- 172.21.15.102:0/375837738 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03601042a0 con 0x7f03601067a0 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.907+0000 7f036532a700 1 --2- 172.21.15.102:0/375837738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f0360106bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0350009e80 tx=0x7f035000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9b3f65b4540528 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:58.698 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.907+0000 7f035ffff700 1 -- 172.21.15.102:0/375837738 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0350017070 con 0x7f03601067a0 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.907+0000 7f035ffff700 1 -- 172.21.15.102:0/375837738 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f035000b8c0 con 0x7f03601067a0 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.907+0000 7f035ffff700 1 -- 172.21.15.102:0/375837738 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0350004470 con 0x7f03601067a0 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 -- 172.21.15.102:0/375837738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 msgr2=0x7f0360106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 --2- 172.21.15.102:0/375837738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f0360106bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0350009e80 tx=0x7f035000b330 comp rx=0 tx=0).stop 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 -- 172.21.15.102:0/375837738 shutdown_connections 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 --2- 172.21.15.102:0/375837738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f0360106bc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 -- 172.21.15.102:0/375837738 >> 172.21.15.102:0/375837738 conn(0x7f03600ffe60 msgr2=0x7f0360102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 -- 172.21.15.102:0/375837738 shutdown_connections 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 -- 172.21.15.102:0/375837738 wait complete. 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 Processor -- start 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 -- start start 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f036019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.908+0000 7f036758e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0360105300 con 0x7f03601067a0 2024-03-30T13:51:58.699 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f036532a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f036019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f036532a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f036019c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60406/0 (socket says 172.21.15.102:60406) 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f036532a700 1 -- 172.21.15.102:0/3346830205 learned_addr learned my addr 172.21.15.102:0/3346830205 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f036532a700 1 -- 172.21.15.102:0/3346830205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036019c850 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f036532a700 1 --2- 172.21.15.102:0/3346830205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f036019c310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0350009fd0 tx=0x7f0350003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f035e7fc700 1 -- 172.21.15.102:0/3346830205 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0350021020 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f035e7fc700 1 -- 172.21.15.102:0/3346830205 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f035000ba70 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036019cae0 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f035e7fc700 1 -- 172.21.15.102:0/3346830205 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0350017040 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.909+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f036019f7d0 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.910+0000 7f035e7fc700 1 -- 172.21.15.102:0/3346830205 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f0350024020 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.910+0000 7f035e7fc700 1 --2- 172.21.15.102:0/3346830205 >> v2:172.21.15.102:6800/3737793096 conn(0x7f0348038dc0 0x7f034803b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.910+0000 7f035e7fc700 1 -- 172.21.15.102:0/3346830205 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f035000f050 con 0x7f03601067a0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.910+0000 7f0364b29700 1 --2- 172.21.15.102:0/3346830205 >> v2:172.21.15.102:6800/3737793096 conn(0x7f0348038dc0 0x7f034803b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:51:58.700 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.910+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0360063870 con 0x7f03601067a0 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.911+0000 7f0364b29700 1 --2- 172.21.15.102:0/3346830205 >> v2:172.21.15.102:6800/3737793096 conn(0x7f0348038dc0 0x7f034803b280 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0354006fd0 tx=0x7f0354006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:57.915+0000 7f035e7fc700 1 -- 172.21.15.102:0/3346830205 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f035002b020 con 0x7f03601067a0 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.114+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f036019fb10 con 0x7f03601067a0 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.117+0000 7f035e7fc700 1 -- 172.21.15.102:0/3346830205 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f0350018280 con 0x7f03601067a0 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 >> v2:172.21.15.102:6800/3737793096 conn(0x7f0348038dc0 msgr2=0x7f034803b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 --2- 172.21.15.102:0/3346830205 >> v2:172.21.15.102:6800/3737793096 conn(0x7f0348038dc0 0x7f034803b280 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0354006fd0 tx=0x7f0354006e40 comp rx=0 tx=0).stop 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 msgr2=0x7f036019c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 --2- 172.21.15.102:0/3346830205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f036019c310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0350009fd0 tx=0x7f0350003ce0 comp rx=0 tx=0).stop 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 shutdown_connections 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 --2- 172.21.15.102:0/3346830205 >> v2:172.21.15.102:6800/3737793096 conn(0x7f0348038dc0 0x7f034803b280 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 --2- 172.21.15.102:0/3346830205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03601067a0 0x7f036019c310 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 >> 172.21.15.102:0/3346830205 conn(0x7f03600ffe60 msgr2=0x7f03601929f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:51:58.701 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 shutdown_connections 2024-03-30T13:51:58.702 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:58.119+0000 7f036758e700 1 -- 172.21.15.102:0/3346830205 wait complete. 2024-03-30T13:51:58.702 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling the dashboard module... 2024-03-30T13:51:59.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:51:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3346830205' entity='client.admin' 2024-03-30T13:52:00.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4024557349' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-30T13:52:00.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:00 smithi102 ceph-mon[29392]: from='mgr.14120 172.21.15.102:0/1970535175' entity='mgr.y' 2024-03-30T13:52:00.443 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:00 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ignoring --setuser ceph since I am not root 2024-03-30T13:52:00.443 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:00 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ignoring --setgroup ceph since I am not root 2024-03-30T13:52:00.760 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:00 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:00.687+0000 7f25836fe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:52:00.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.095+0000 7f7d40b4f700 1 Processor -- start 2024-03-30T13:52:00.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.095+0000 7f7d40b4f700 1 -- start start 2024-03-30T13:52:00.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.095+0000 7f7d40b4f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:00.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.095+0000 7f7d40b4f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c079680 con 0x7f7d3c106dc0 2024-03-30T13:52:00.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.095+0000 7f7d3a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:00.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.095+0000 7f7d3a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60418/0 (socket says 172.21.15.102:60418) 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.095+0000 7f7d3a59c700 1 -- 172.21.15.102:0/2937193098 learned_addr learned my addr 172.21.15.102:0/2937193098 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.096+0000 7f7d3a59c700 1 -- 172.21.15.102:0/2937193098 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c1096f0 con 0x7f7d3c106dc0 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.096+0000 7f7d3a59c700 1 --2- 172.21.15.102:0/2937193098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c1091b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098a0 tx=0x7f7d2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287fd3351e07d929 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.096+0000 7f7d3959a700 1 -- 172.21.15.102:0/2937193098 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d2c004160 con 0x7f7d3c106dc0 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.096+0000 7f7d3959a700 1 -- 172.21.15.102:0/2937193098 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f7d2c004300 con 0x7f7d3c106dc0 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.096+0000 7f7d3959a700 1 -- 172.21.15.102:0/2937193098 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d2c011a00 con 0x7f7d3c106dc0 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 -- 172.21.15.102:0/2937193098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 msgr2=0x7f7d3c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 --2- 172.21.15.102:0/2937193098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c1091b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0098a0 tx=0x7f7d2c009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 -- 172.21.15.102:0/2937193098 shutdown_connections 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 --2- 172.21.15.102:0/2937193098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c1091b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 -- 172.21.15.102:0/2937193098 >> 172.21.15.102:0/2937193098 conn(0x7f7d3c100790 msgr2=0x7f7d3c102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 -- 172.21.15.102:0/2937193098 shutdown_connections 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 -- 172.21.15.102:0/2937193098 wait complete. 2024-03-30T13:52:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.097+0000 7f7d40b4f700 1 Processor -- start 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d40b4f700 1 -- start start 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d40b4f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d40b4f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c10a630 con 0x7f7d3c106dc0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d3a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d3a59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60428/0 (socket says 172.21.15.102:60428) 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d3a59c700 1 -- 172.21.15.102:0/4024557349 learned_addr learned my addr 172.21.15.102:0/4024557349 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d3a59c700 1 -- 172.21.15.102:0/4024557349 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c19a690 con 0x7f7d3c106dc0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d3a59c700 1 --2- 172.21.15.102:0/4024557349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c19a150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c011a10 tx=0x7f7d2c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d2c01a430 con 0x7f7d3c106dc0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f7d2c01aa80 con 0x7f7d3c106dc0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c19a920 con 0x7f7d3c106dc0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d2c0249d0 con 0x7f7d3c106dc0 2024-03-30T13:52:00.770 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.098+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c19d610 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.099+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46081+0+0 (secure 0 0 0) 0x7f7d2c022020 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.099+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d3c063870 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.100+0000 7f7d2b7fe700 1 --2- 172.21.15.102:0/4024557349 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7d24038dc0 0x7f7d2403b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.100+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f7d2c04cf20 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.100+0000 7f7d39d9b700 1 --2- 172.21.15.102:0/4024557349 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7d24038dc0 0x7f7d2403b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.100+0000 7f7d39d9b700 1 --2- 172.21.15.102:0/4024557349 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7d24038dc0 0x7f7d2403b280 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d30006fd0 tx=0x7f7d30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.105+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7d2c00b3e0 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:51:59.360+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f7d3c19d9f0 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.120+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f7d2c04cc80 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.122+0000 7f7d2b7fe700 1 -- 172.21.15.102:0/4024557349 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 9) ==== 46094+0+0 (secure 0 0 0) 0x7f7d2c02dc90 con 0x7f7d3c106dc0 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7d24038dc0 msgr2=0x7f7d2403b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 --2- 172.21.15.102:0/4024557349 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7d24038dc0 0x7f7d2403b280 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d30006fd0 tx=0x7f7d30006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 msgr2=0x7f7d3c19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:00.771 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 --2- 172.21.15.102:0/4024557349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c19a150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c011a10 tx=0x7f7d2c011af0 comp rx=0 tx=0).stop 2024-03-30T13:52:00.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 shutdown_connections 2024-03-30T13:52:00.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 --2- 172.21.15.102:0/4024557349 >> v2:172.21.15.102:6800/3737793096 conn(0x7f7d24038dc0 0x7f7d2403b280 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:00.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 --2- 172.21.15.102:0/4024557349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d3c106dc0 0x7f7d3c19a150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:00.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 >> 172.21.15.102:0/4024557349 conn(0x7f7d3c100790 msgr2=0x7f7d3c101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:00.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 shutdown_connections 2024-03-30T13:52:00.772 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:00.123+0000 7f7d40b4f700 1 -- 172.21.15.102:0/4024557349 wait complete. 2024-03-30T13:52:01.012 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:00 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:00.792+0000 7f25836fe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:52:01.412 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4024557349' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-30T13:52:01.412 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:01 smithi102 ceph-mon[29392]: mgrmap e9: y(active, since 19s) 2024-03-30T13:52:01.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:01 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:01.452+0000 7f25836fe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:52:01.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:01 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:01.550+0000 7f25836fe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:52:02.075 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:01 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:01.695+0000 7f25836fe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:52:02.075 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:01 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:01.792+0000 7f25836fe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:52:02.075 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:01 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:01.877+0000 7f25836fe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:52:02.083 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca7ad3700 1 Processor -- start 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca7ad3700 1 -- start start 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca7ad3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca00733a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca7ad3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca010e4c0 con 0x7f9ca010e010 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca586f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca00733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca586f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca00733a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59292/0 (socket says 172.21.15.102:59292) 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca586f700 1 -- 172.21.15.102:0/165308712 learned_addr learned my addr 172.21.15.102:0/165308712 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:02.084 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.206+0000 7f9ca586f700 1 -- 172.21.15.102:0/165308712 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca00738e0 con 0x7f9ca010e010 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.207+0000 7f9ca586f700 1 --2- 172.21.15.102:0/165308712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca00733a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0098a0 tx=0x7f9c9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680bc23497c8b3a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.207+0000 7f9ca486d700 1 -- 172.21.15.102:0/165308712 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c9c004160 con 0x7f9ca010e010 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.207+0000 7f9ca486d700 1 -- 172.21.15.102:0/165308712 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f9c9c004300 con 0x7f9ca010e010 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.207+0000 7f9ca486d700 1 -- 172.21.15.102:0/165308712 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c9c011a00 con 0x7f9ca010e010 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/165308712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 msgr2=0x7f9ca00733a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 --2- 172.21.15.102:0/165308712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca00733a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0098a0 tx=0x7f9c9c009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/165308712 shutdown_connections 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 --2- 172.21.15.102:0/165308712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca00733a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/165308712 >> 172.21.15.102:0/165308712 conn(0x7f9ca006ec10 msgr2=0x7f9ca0071050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/165308712 shutdown_connections 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/165308712 wait complete. 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 Processor -- start 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.208+0000 7f9ca7ad3700 1 -- start start 2024-03-30T13:52:02.085 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca7ad3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca01101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca7ad3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0112e00 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca586f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca01101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca586f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca01101f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59304/0 (socket says 172.21.15.102:59304) 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca586f700 1 -- 172.21.15.102:0/1748866326 learned_addr learned my addr 172.21.15.102:0/1748866326 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca586f700 1 -- 172.21.15.102:0/1748866326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca0110730 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca586f700 1 --2- 172.21.15.102:0/1748866326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca01101f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c011a10 tx=0x7f9c9c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9c96ffd700 1 -- 172.21.15.102:0/1748866326 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c9c003aa0 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9c96ffd700 1 -- 172.21.15.102:0/1748866326 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f9c9c01a8e0 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca010e920 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9c96ffd700 1 -- 172.21.15.102:0/1748866326 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c9c024900 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.209+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca010ee60 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.210+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca0063870 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.210+0000 7f9c96ffd700 1 -- 172.21.15.102:0/1748866326 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) ==== 46094+0+0 (secure 0 0 0) 0x7f9c9c022020 con 0x7f9ca010e010 2024-03-30T13:52:02.086 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.211+0000 7f9c96ffd700 1 --2- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 0x7f9c8c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.211+0000 7f9c96ffd700 1 -- 172.21.15.102:0/1748866326 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7f9c9c04cd30 con 0x7f9ca010e010 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.211+0000 7f9ca506e700 1 -- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 msgr2=0x7f9c8c03b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3737793096 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.211+0000 7f9ca506e700 1 --2- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 0x7f9c8c03b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.215+0000 7f9c96ffd700 1 -- 172.21.15.102:0/1748866326 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c9c00b3e0 con 0x7f9ca010e010 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.412+0000 7f9ca506e700 1 -- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 msgr2=0x7f9c8c03b2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3737793096 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.412+0000 7f9ca506e700 1 --2- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 0x7f9c8c03b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.473+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f9ca010f550 con 0x7f9ca010e010 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.473+0000 7f9c96ffd700 1 -- 172.21.15.102:0/1748866326 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f9c9c00fdd0 con 0x7f9ca010e010 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 msgr2=0x7f9c8c03b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 --2- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 0x7f9c8c03b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 msgr2=0x7f9ca01101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 --2- 172.21.15.102:0/1748866326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca01101f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c011a10 tx=0x7f9c9c011af0 comp rx=0 tx=0).stop 2024-03-30T13:52:02.087 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 shutdown_connections 2024-03-30T13:52:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 --2- 172.21.15.102:0/1748866326 >> v2:172.21.15.102:6800/3737793096 conn(0x7f9c8c038e10 0x7f9c8c03b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 --2- 172.21.15.102:0/1748866326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca010e010 0x7f9ca01101f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 >> 172.21.15.102:0/1748866326 conn(0x7f9ca006ec10 msgr2=0x7f9ca006f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 shutdown_connections 2024-03-30T13:52:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:01.476+0000 7f9ca7ad3700 1 -- 172.21.15.102:0/1748866326 wait complete. 2024-03-30T13:52:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-03-30T13:52:02.088 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 9... 2024-03-30T13:52:02.353 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1748866326' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T13:52:02.354 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:02 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:02.148+0000 7f25836fe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:52:02.354 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:02 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:02.260+0000 7f25836fe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:52:02.354 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:02 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:02.353+0000 7f25836fe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:52:02.635 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:02 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:02.455+0000 7f25836fe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:52:02.635 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:02 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:02.634+0000 7f25836fe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:52:03.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:03 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:03.594+0000 7f25836fe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:52:03.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:03 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:03.723+0000 7f25836fe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:52:04.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:04 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:04.056+0000 7f25836fe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:52:04.192 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:04 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:04.146+0000 7f25836fe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:52:04.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:04 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:04.341+0000 7f25836fe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:52:04.692 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:04 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:04.525+0000 7f25836fe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:52:05.066 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:04 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:04.809+0000 7f25836fe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:52:05.066 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:05 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:05.065+0000 7f25836fe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:52:05.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:05 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:05.302+0000 7f25836fe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:52:06.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:06.104+0000 7f25836fe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:52:06.942 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:06.669+0000 7f25836fe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:52:06.942 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:06.769+0000 7f25836fe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:52:07.442 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:52:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:52:06.988+0000 7f25836fe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:52:07.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: Active manager daemon y restarted 2024-03-30T13:52:07.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: Activating manager daemon y 2024-03-30T13:52:07.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: osdmap e3: 0 total, 0 up, 0 in 2024-03-30T13:52:07.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: mgrmap e10: y(active, starting, since 0.00320593s) 2024-03-30T13:52:07.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: Manager daemon y is now available 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:52:07.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.512+0000 7fbd095ab700 1 Processor -- start 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.512+0000 7fbd095ab700 1 -- start start 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.512+0000 7fbd095ab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd04106b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.512+0000 7fbd095ab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbd04107120 con 0x7fbd04106730 2024-03-30T13:52:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.512+0000 7fbd02ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd04106b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.512+0000 7fbd02ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd04106b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59314/0 (socket says 172.21.15.102:59314) 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.512+0000 7fbd02ffd700 1 -- 172.21.15.102:0/4104232322 learned_addr learned my addr 172.21.15.102:0/4104232322 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.513+0000 7fbd02ffd700 1 -- 172.21.15.102:0/4104232322 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd041072a0 con 0x7fbd04106730 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.513+0000 7fbd02ffd700 1 --2- 172.21.15.102:0/4104232322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd04106b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80098a0 tx=0x7fbcf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f343b79ffbd307 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.513+0000 7fbd01ffb700 1 -- 172.21.15.102:0/4104232322 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf8004160 con 0x7fbd04106730 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.513+0000 7fbd01ffb700 1 -- 172.21.15.102:0/4104232322 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fbcf8004300 con 0x7fbd04106730 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.513+0000 7fbd01ffb700 1 -- 172.21.15.102:0/4104232322 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf8011a00 con 0x7fbd04106730 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.514+0000 7fbd095ab700 1 -- 172.21.15.102:0/4104232322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 msgr2=0x7fbd04106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.514+0000 7fbd095ab700 1 --2- 172.21.15.102:0/4104232322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd04106b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80098a0 tx=0x7fbcf8009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.514+0000 7fbd095ab700 1 -- 172.21.15.102:0/4104232322 shutdown_connections 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.514+0000 7fbd095ab700 1 --2- 172.21.15.102:0/4104232322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd04106b50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.514+0000 7fbd095ab700 1 -- 172.21.15.102:0/4104232322 >> 172.21.15.102:0/4104232322 conn(0x7fbd040ffec0 msgr2=0x7fbd041022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.514+0000 7fbd095ab700 1 -- 172.21.15.102:0/4104232322 shutdown_connections 2024-03-30T13:52:08.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.514+0000 7fbd095ab700 1 -- 172.21.15.102:0/4104232322 wait complete. 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd095ab700 1 Processor -- start 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd095ab700 1 -- start start 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd095ab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd0419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd095ab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041050f0 con 0x7fbd04106730 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd02ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd0419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd02ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd0419c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59320/0 (socket says 172.21.15.102:59320) 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd02ffd700 1 -- 172.21.15.102:0/4136243496 learned_addr learned my addr 172.21.15.102:0/4136243496 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd02ffd700 1 -- 172.21.15.102:0/4136243496 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0419c830 con 0x7fbd04106730 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.515+0000 7fbd02ffd700 1 --2- 172.21.15.102:0/4136243496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd0419c2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8011a10 tx=0x7fbcf8011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.516+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf8003aa0 con 0x7fbd04106730 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.516+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0419cb20 con 0x7fbd04106730 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.516+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fbcf801a8e0 con 0x7fbd04106730 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.516+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcf8024900 con 0x7fbd04106730 2024-03-30T13:52:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.516+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0419f760 con 0x7fbd04106730 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.517+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) ==== 46094+0+0 (secure 0 0 0) 0x7fbcf8022020 con 0x7fbd04106730 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.517+0000 7fbcebfff700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.517+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 --> v2:172.21.15.102:6800/3737793096 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbcec03b950 con 0x7fbcec038dc0 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.517+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (secure 0 0 0) 0x7fbcf804d290 con 0x7fbd04106730 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.517+0000 7fbd027fc700 1 -- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 msgr2=0x7fbcec03b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3737793096 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.517+0000 7fbd027fc700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.718+0000 7fbd027fc700 1 -- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 msgr2=0x7fbcec03b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3737793096 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:02.718+0000 7fbd027fc700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:03.119+0000 7fbd027fc700 1 -- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 msgr2=0x7fbcec03b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3737793096 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:03.119+0000 7fbd027fc700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:03.920+0000 7fbd027fc700 1 -- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 msgr2=0x7fbcec03b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3737793096 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:03.920+0000 7fbd027fc700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:05.521+0000 7fbd027fc700 1 -- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 msgr2=0x7fbcec03b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3737793096 2024-03-30T13:52:08.989 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:05.521+0000 7fbd027fc700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:06.991+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 10) ==== 45896+0+0 (secure 0 0 0) 0x7fbcf80193c0 con 0x7fbd04106730 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:06.991+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 msgr2=0x7fbcec03b280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:06.991+0000 7fbcebfff700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:07.993+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 11) ==== 45988+0+0 (secure 0 0 0) 0x7fbcf804e9e0 con 0x7fbd04106730 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:07.993+0000 7fbcebfff700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbcec03c810 0x7fbcec03ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:07.993+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 --> v2:172.21.15.102:6800/3154847229 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbcf8009530 con 0x7fbcec03c810 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:07.993+0000 7fbd027fc700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbcec03c810 0x7fbcec03ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:07.994+0000 7fbd027fc700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbcec03c810 0x7fbcec03ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4003a10 tx=0x7fbcf40092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:07.994+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fbcf8009530 con 0x7fbcec03c810 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.001+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 --> v2:172.21.15.102:6800/3154847229 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fbd04063870 con 0x7fbcec03c810 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.001+0000 7fbcebfff700 1 -- 172.21.15.102:0/4136243496 <== mgr.14152 v2:172.21.15.102:6800/3154847229 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fbd04063870 con 0x7fbcec03c810 2024-03-30T13:52:08.990 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbcec03c810 msgr2=0x7fbcec03ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbcec03c810 0x7fbcec03ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4003a10 tx=0x7fbcf40092b0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 msgr2=0x7fbd0419c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 --2- 172.21.15.102:0/4136243496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd0419c2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8011a10 tx=0x7fbcf8011af0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 shutdown_connections 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbcec03c810 0x7fbcec03ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 --2- 172.21.15.102:0/4136243496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd04106730 0x7fbd0419c2f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 --2- 172.21.15.102:0/4136243496 >> v2:172.21.15.102:6800/3737793096 conn(0x7fbcec038dc0 0x7fbcec03b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 >> 172.21.15.102:0/4136243496 conn(0x7fbd040ffec0 msgr2=0x7fbd04192910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 shutdown_connections 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:08.002+0000 7fbd095ab700 1 -- 172.21.15.102:0/4136243496 wait complete. 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 9 is available 2024-03-30T13:52:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:Generating a dashboard self-signed certificate... 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: [30/Mar/2024:13:52:07] ENGINE Bus STARTING 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: [30/Mar/2024:13:52:07] ENGINE Serving on http://172.21.15.102:8765 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: mgrmap e11: y(active, since 1.00541s) 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: [30/Mar/2024:13:52:08] ENGINE Serving on https://172.21.15.102:7150 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: [30/Mar/2024:13:52:08] ENGINE Bus STARTED 2024-03-30T13:52:09.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:10.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-30T13:52:10.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f1724849700 1 Processor -- start 2024-03-30T13:52:10.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f1724849700 1 -- start start 2024-03-30T13:52:10.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f1724849700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720105590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:10.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f1724849700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1720105b60 con 0x7f1720105170 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f171e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720105590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f171e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720105590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59408/0 (socket says 172.21.15.102:59408) 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f171e59c700 1 -- 172.21.15.102:0/1703799927 learned_addr learned my addr 172.21.15.102:0/1703799927 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.432+0000 7f171e59c700 1 -- 172.21.15.102:0/1703799927 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1720106370 con 0x7f1720105170 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.433+0000 7f171e59c700 1 --2- 172.21.15.102:0/1703799927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720105590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f17100098a0 tx=0x7f1710009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31fd71752489c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.433+0000 7f171d59a700 1 -- 172.21.15.102:0/1703799927 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1710004160 con 0x7f1720105170 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.433+0000 7f171d59a700 1 -- 172.21.15.102:0/1703799927 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f1710004300 con 0x7f1720105170 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.433+0000 7f171d59a700 1 -- 172.21.15.102:0/1703799927 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1710011a00 con 0x7f1720105170 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.433+0000 7f1724849700 1 -- 172.21.15.102:0/1703799927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 msgr2=0x7f1720105590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.433+0000 7f1724849700 1 --2- 172.21.15.102:0/1703799927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720105590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f17100098a0 tx=0x7f1710009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.434+0000 7f1724849700 1 -- 172.21.15.102:0/1703799927 shutdown_connections 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.434+0000 7f1724849700 1 --2- 172.21.15.102:0/1703799927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720105590 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.434+0000 7f1724849700 1 -- 172.21.15.102:0/1703799927 >> 172.21.15.102:0/1703799927 conn(0x7f1720100730 msgr2=0x7f1720102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.434+0000 7f1724849700 1 -- 172.21.15.102:0/1703799927 shutdown_connections 2024-03-30T13:52:10.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.434+0000 7f1724849700 1 -- 172.21.15.102:0/1703799927 wait complete. 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.434+0000 7f1724849700 1 Processor -- start 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.434+0000 7f1724849700 1 -- start start 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f1724849700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720079bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f1724849700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1720109720 con 0x7f1720105170 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f171e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720079bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f171e59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720079bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59420/0 (socket says 172.21.15.102:59420) 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f171e59c700 1 -- 172.21.15.102:0/2616138856 learned_addr learned my addr 172.21.15.102:0/2616138856 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f171e59c700 1 -- 172.21.15.102:0/2616138856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172019ced0 con 0x7f1720105170 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f171e59c700 1 --2- 172.21.15.102:0/2616138856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720079bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1710009fb0 tx=0x7f1710011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f170f7fe700 1 -- 172.21.15.102:0/2616138856 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1710003aa0 con 0x7f1720105170 2024-03-30T13:52:10.317 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172019d0a0 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f170f7fe700 1 -- 172.21.15.102:0/2616138856 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f171001a8e0 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.435+0000 7f170f7fe700 1 -- 172.21.15.102:0/2616138856 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1710024930 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.436+0000 7f170f7fe700 1 -- 172.21.15.102:0/2616138856 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) ==== 46094+0+0 (secure 0 0 0) 0x7f1710022020 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.436+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f172019d4c0 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.436+0000 7f170f7fe700 1 --2- 172.21.15.102:0/2616138856 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1708038dc0 0x7f170803b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.436+0000 7f170f7fe700 1 -- 172.21.15.102:0/2616138856 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7f1710053170 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.437+0000 7f171dd9b700 1 --2- 172.21.15.102:0/2616138856 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1708038dc0 0x7f170803b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.437+0000 7f171dd9b700 1 --2- 172.21.15.102:0/2616138856 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1708038dc0 0x7f170803b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1714006fd0 tx=0x7f1714006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.437+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1720063870 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.442+0000 7f170f7fe700 1 -- 172.21.15.102:0/2616138856 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f171001ad70 con 0x7f1720105170 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.649+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f17201018e0 con 0x7f1708038dc0 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.733+0000 7f170f7fe700 1 -- 172.21.15.102:0/2616138856 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f17201018e0 con 0x7f1708038dc0 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1708038dc0 msgr2=0x7f170803b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:10.318 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 --2- 172.21.15.102:0/2616138856 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1708038dc0 0x7f170803b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1714006fd0 tx=0x7f1714006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 msgr2=0x7f1720079bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 --2- 172.21.15.102:0/2616138856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720079bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1710009fb0 tx=0x7f1710011ac0 comp rx=0 tx=0).stop 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 shutdown_connections 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 --2- 172.21.15.102:0/2616138856 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1708038dc0 0x7f170803b280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 --2- 172.21.15.102:0/2616138856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1720105170 0x7f1720079bf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 >> 172.21.15.102:0/2616138856 conn(0x7f1720100730 msgr2=0x7f17201011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 shutdown_connections 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:09.736+0000 7f1724849700 1 -- 172.21.15.102:0/2616138856 wait complete. 2024-03-30T13:52:10.319 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial admin user... 2024-03-30T13:52:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:10 smithi102 ceph-mon[29392]: mgrmap e12: y(active, since 2s) 2024-03-30T13:52:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:10 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:10.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:10 smithi102 ceph-mon[29392]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:52:10.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:10 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:10.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:10 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:11.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:11 smithi102 ceph-mon[29392]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:52:11.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:11 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$5JOLYDTYkUUyYLLNSyeFrO0OsPozIyDf0q5iNMCJa/EqDtSTiaYIS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711806731, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.727+0000 7f4409585700 1 Processor -- start 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.727+0000 7f4409585700 1 -- start start 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.727+0000 7f4409585700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404109100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.727+0000 7f4409585700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4404079720 con 0x7f4404106d10 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51606/0 (socket says 172.21.15.102:51606) 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4402ffd700 1 -- 172.21.15.102:0/478520794 learned_addr learned my addr 172.21.15.102:0/478520794 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:12.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4402ffd700 1 -- 172.21.15.102:0/478520794 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4404109640 con 0x7f4404106d10 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4402ffd700 1 --2- 172.21.15.102:0/478520794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404109100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f43f40098a0 tx=0x7f43f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbcfddd1289c277e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4401ffb700 1 -- 172.21.15.102:0/478520794 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43f4004160 con 0x7f4404106d10 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4401ffb700 1 -- 172.21.15.102:0/478520794 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f43f4004300 con 0x7f4404106d10 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.728+0000 7f4401ffb700 1 -- 172.21.15.102:0/478520794 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43f4011a00 con 0x7f4404106d10 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.729+0000 7f4409585700 1 -- 172.21.15.102:0/478520794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 msgr2=0x7f4404109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.729+0000 7f4409585700 1 --2- 172.21.15.102:0/478520794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404109100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f43f40098a0 tx=0x7f43f4009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.729+0000 7f4409585700 1 -- 172.21.15.102:0/478520794 shutdown_connections 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.729+0000 7f4409585700 1 --2- 172.21.15.102:0/478520794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404109100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.729+0000 7f4409585700 1 -- 172.21.15.102:0/478520794 >> 172.21.15.102:0/478520794 conn(0x7f4404100740 msgr2=0x7f4404102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.729+0000 7f4409585700 1 -- 172.21.15.102:0/478520794 shutdown_connections 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.729+0000 7f4409585700 1 -- 172.21.15.102:0/478520794 wait complete. 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4409585700 1 Processor -- start 2024-03-30T13:52:12.339 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4409585700 1 -- start start 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4409585700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404106590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4409585700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f440410a640 con 0x7f4404106d10 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404106590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404106590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51614/0 (socket says 172.21.15.102:51614) 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4402ffd700 1 -- 172.21.15.102:0/4186232722 learned_addr learned my addr 172.21.15.102:0/4186232722 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.730+0000 7f4402ffd700 1 -- 172.21.15.102:0/4186232722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4404106ad0 con 0x7f4404106d10 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.731+0000 7f4402ffd700 1 --2- 172.21.15.102:0/4186232722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404106590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43f4011a10 tx=0x7f43f4011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.731+0000 7f43ebfff700 1 -- 172.21.15.102:0/4186232722 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43f4003aa0 con 0x7f4404106d10 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.731+0000 7f43ebfff700 1 -- 172.21.15.102:0/4186232722 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f43f401a8e0 con 0x7f4404106d10 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.731+0000 7f43ebfff700 1 -- 172.21.15.102:0/4186232722 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43f4024900 con 0x7f4404106d10 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.731+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4404104cc0 con 0x7f4404106d10 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.731+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4404105200 con 0x7f4404106d10 2024-03-30T13:52:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.732+0000 7f43ebfff700 1 -- 172.21.15.102:0/4186232722 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) ==== 46094+0+0 (secure 0 0 0) 0x7f43f4022020 con 0x7f4404106d10 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.732+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4404063870 con 0x7f4404106d10 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.732+0000 7f43ebfff700 1 --2- 172.21.15.102:0/4186232722 >> v2:172.21.15.102:6800/3154847229 conn(0x7f43ec038e10 0x7f43ec03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.732+0000 7f43ebfff700 1 -- 172.21.15.102:0/4186232722 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7f43f404d110 con 0x7f4404106d10 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.732+0000 7f44027fc700 1 --2- 172.21.15.102:0/4186232722 >> v2:172.21.15.102:6800/3154847229 conn(0x7f43ec038e10 0x7f43ec03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.733+0000 7f44027fc700 1 --2- 172.21.15.102:0/4186232722 >> v2:172.21.15.102:6800/3154847229 conn(0x7f43ec038e10 0x7f43ec03b2d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f43f8006fd0 tx=0x7f43f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.738+0000 7f43ebfff700 1 -- 172.21.15.102:0/4186232722 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f43f400fdd0 con 0x7f4404106d10 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:10.968+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f44041058f0 con 0x7f43ec038e10 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.193+0000 7f43ebfff700 1 -- 172.21.15.102:0/4186232722 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f44041058f0 con 0x7f43ec038e10 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 >> v2:172.21.15.102:6800/3154847229 conn(0x7f43ec038e10 msgr2=0x7f43ec03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 --2- 172.21.15.102:0/4186232722 >> v2:172.21.15.102:6800/3154847229 conn(0x7f43ec038e10 0x7f43ec03b2d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f43f8006fd0 tx=0x7f43f8006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 msgr2=0x7f4404106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:12.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 --2- 172.21.15.102:0/4186232722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404106590 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43f4011a10 tx=0x7f43f4011af0 comp rx=0 tx=0).stop 2024-03-30T13:52:12.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 shutdown_connections 2024-03-30T13:52:12.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 --2- 172.21.15.102:0/4186232722 >> v2:172.21.15.102:6800/3154847229 conn(0x7f43ec038e10 0x7f43ec03b2d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:12.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 --2- 172.21.15.102:0/4186232722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404106d10 0x7f4404106590 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:12.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 >> 172.21.15.102:0/4186232722 conn(0x7f4404100740 msgr2=0x7f4404101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:12.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 shutdown_connections 2024-03-30T13:52:12.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:11.196+0000 7f4409585700 1 -- 172.21.15.102:0/4186232722 wait complete. 2024-03-30T13:52:12.342 INFO:teuthology.orchestra.run.smithi102.stdout:Fetching dashboard port number... 2024-03-30T13:52:13.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3240493260' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 8443 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.758+0000 7f616a537700 1 Processor -- start 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.758+0000 7f616a537700 1 -- start start 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.758+0000 7f616a537700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f6164106660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.758+0000 7f616a537700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6164106c30 con 0x7f6164106240 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.758+0000 7f6163fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f6164106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.758+0000 7f6163fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f6164106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51622/0 (socket says 172.21.15.102:51622) 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.758+0000 7f6163fff700 1 -- 172.21.15.102:0/3337019430 learned_addr learned my addr 172.21.15.102:0/3337019430 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:13.544 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.759+0000 7f6163fff700 1 -- 172.21.15.102:0/3337019430 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6164107440 con 0x7f6164106240 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.759+0000 7f6163fff700 1 --2- 172.21.15.102:0/3337019430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f6164106660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f61540098a0 tx=0x7f6154009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49239a695c4ff09e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.759+0000 7f6162ffd700 1 -- 172.21.15.102:0/3337019430 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6154004160 con 0x7f6164106240 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.759+0000 7f6162ffd700 1 -- 172.21.15.102:0/3337019430 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f6154004300 con 0x7f6164106240 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.759+0000 7f6162ffd700 1 -- 172.21.15.102:0/3337019430 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6154011a00 con 0x7f6164106240 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.759+0000 7f616a537700 1 -- 172.21.15.102:0/3337019430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 msgr2=0x7f6164106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.759+0000 7f616a537700 1 --2- 172.21.15.102:0/3337019430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f6164106660 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f61540098a0 tx=0x7f6154009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 -- 172.21.15.102:0/3337019430 shutdown_connections 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 --2- 172.21.15.102:0/3337019430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f6164106660 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 -- 172.21.15.102:0/3337019430 >> 172.21.15.102:0/3337019430 conn(0x7f6164101800 msgr2=0x7f6164103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 -- 172.21.15.102:0/3337019430 shutdown_connections 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 -- 172.21.15.102:0/3337019430 wait complete. 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 Processor -- start 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 -- start start 2024-03-30T13:52:13.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.760+0000 7f616a537700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f616419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f616a537700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f616410a7f0 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f6163fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f616419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f6163fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f616419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51632/0 (socket says 172.21.15.102:51632) 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f6163fff700 1 -- 172.21.15.102:0/3240493260 learned_addr learned my addr 172.21.15.102:0/3240493260 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f6163fff700 1 -- 172.21.15.102:0/3240493260 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616419a650 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f6163fff700 1 --2- 172.21.15.102:0/3240493260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f616419a110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6154009fb0 tx=0x7f6154011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f61617fa700 1 -- 172.21.15.102:0/3240493260 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f615401a430 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f61617fa700 1 -- 172.21.15.102:0/3240493260 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f615401aa80 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616419a8e0 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f61617fa700 1 -- 172.21.15.102:0/3240493260 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61540249b0 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.761+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616419d5d0 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.762+0000 7f61617fa700 1 -- 172.21.15.102:0/3240493260 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) ==== 46094+0+0 (secure 0 0 0) 0x7f6154022020 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.762+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6164063870 con 0x7f6164106240 2024-03-30T13:52:13.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.762+0000 7f61617fa700 1 --2- 172.21.15.102:0/3240493260 >> v2:172.21.15.102:6800/3154847229 conn(0x7f614c038dc0 0x7f614c03b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.762+0000 7f61617fa700 1 -- 172.21.15.102:0/3240493260 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7f615404d5a0 con 0x7f6164106240 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.763+0000 7f61637fe700 1 --2- 172.21.15.102:0/3240493260 >> v2:172.21.15.102:6800/3154847229 conn(0x7f614c038dc0 0x7f614c03b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.763+0000 7f61637fe700 1 --2- 172.21.15.102:0/3240493260 >> v2:172.21.15.102:6800/3154847229 conn(0x7f614c038dc0 0x7f614c03b280 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6158006fd0 tx=0x7f6158006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.767+0000 7f61617fa700 1 -- 172.21.15.102:0/3240493260 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f615400fdd0 con 0x7f6164106240 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.962+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f616419d9b0 con 0x7f6164106240 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.962+0000 7f61617fa700 1 -- 172.21.15.102:0/3240493260 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f615402d4a0 con 0x7f6164106240 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.964+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 >> v2:172.21.15.102:6800/3154847229 conn(0x7f614c038dc0 msgr2=0x7f614c03b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.964+0000 7f616a537700 1 --2- 172.21.15.102:0/3240493260 >> v2:172.21.15.102:6800/3154847229 conn(0x7f614c038dc0 0x7f614c03b280 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6158006fd0 tx=0x7f6158006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.964+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 msgr2=0x7f616419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.964+0000 7f616a537700 1 --2- 172.21.15.102:0/3240493260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f616419a110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6154009fb0 tx=0x7f6154011ac0 comp rx=0 tx=0).stop 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.965+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 shutdown_connections 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.965+0000 7f616a537700 1 --2- 172.21.15.102:0/3240493260 >> v2:172.21.15.102:6800/3154847229 conn(0x7f614c038dc0 0x7f614c03b280 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.965+0000 7f616a537700 1 --2- 172.21.15.102:0/3240493260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6164106240 0x7f616419a110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:13.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.965+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 >> 172.21.15.102:0/3240493260 conn(0x7f6164101800 msgr2=0x7f6164102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:13.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.965+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 shutdown_connections 2024-03-30T13:52:13.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:12.965+0000 7f616a537700 1 -- 172.21.15.102:0/3240493260 wait complete. 2024-03-30T13:52:13.551 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T13:52:13.551 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-03-30T13:52:13.558 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T13:52:13.558 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-03-30T13:52:13.558 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-03-30T13:52:13.558 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-30T13:52:13.560 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph Dashboard is now available at: 2024-03-30T13:52:13.560 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:13.560 INFO:teuthology.orchestra.run.smithi102.stdout: URL: https://smithi102.front.sepia.ceph.com:8443/ 2024-03-30T13:52:13.560 INFO:teuthology.orchestra.run.smithi102.stdout: User: admin 2024-03-30T13:52:13.560 INFO:teuthology.orchestra.run.smithi102.stdout: Password: ugl8cbjsqd 2024-03-30T13:52:13.560 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:13.560 INFO:teuthology.orchestra.run.smithi102.stdout:Saving cluster configuration to /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config directory 2024-03-30T13:52:13.561 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling autotune for osd_memory_target 2024-03-30T13:52:14.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:14 smithi102 ceph-mon[29392]: mgrmap e13: y(active, since 6s) 2024-03-30T13:52:14.727 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.995+0000 7ffa2f694700 1 Processor -- start 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2f694700 1 -- start start 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2f694700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa28109170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2f694700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28079f30 con 0x7ffa28106d80 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2d430700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa28109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2d430700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa28109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51642/0 (socket says 172.21.15.102:51642) 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2d430700 1 -- 172.21.15.102:0/3277170872 learned_addr learned my addr 172.21.15.102:0/3277170872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2d430700 1 -- 172.21.15.102:0/3277170872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa281096b0 con 0x7ffa28106d80 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.996+0000 7ffa2d430700 1 --2- 172.21.15.102:0/3277170872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa28109170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0098a0 tx=0x7ffa1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f97873a8c02653e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa1bfff700 1 -- 172.21.15.102:0/3277170872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa1c004160 con 0x7ffa28106d80 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa1bfff700 1 -- 172.21.15.102:0/3277170872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7ffa1c004300 con 0x7ffa28106d80 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa1bfff700 1 -- 172.21.15.102:0/3277170872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa1c011a00 con 0x7ffa28106d80 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa2f694700 1 -- 172.21.15.102:0/3277170872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 msgr2=0x7ffa28109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa2f694700 1 --2- 172.21.15.102:0/3277170872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa28109170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0098a0 tx=0x7ffa1c009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:14.728 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa2f694700 1 -- 172.21.15.102:0/3277170872 shutdown_connections 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa2f694700 1 --2- 172.21.15.102:0/3277170872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa28109170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa2f694700 1 -- 172.21.15.102:0/3277170872 >> 172.21.15.102:0/3277170872 conn(0x7ffa281006f0 msgr2=0x7ffa28102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa2f694700 1 -- 172.21.15.102:0/3277170872 shutdown_connections 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.997+0000 7ffa2f694700 1 -- 172.21.15.102:0/3277170872 wait complete. 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2f694700 1 Processor -- start 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2f694700 1 -- start start 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2f694700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa2819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2f694700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2810a600 con 0x7ffa28106d80 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2d430700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa2819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:14.729 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2d430700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa2819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51650/0 (socket says 172.21.15.102:51650) 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2d430700 1 -- 172.21.15.102:0/245945623 learned_addr learned my addr 172.21.15.102:0/245945623 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.998+0000 7ffa2d430700 1 -- 172.21.15.102:0/245945623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2819a660 con 0x7ffa28106d80 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.999+0000 7ffa2d430700 1 --2- 172.21.15.102:0/245945623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa2819a120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c011a10 tx=0x7ffa1c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.999+0000 7ffa1a7fc700 1 -- 172.21.15.102:0/245945623 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa1c01a430 con 0x7ffa28106d80 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.999+0000 7ffa1a7fc700 1 -- 172.21.15.102:0/245945623 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7ffa1c01aa80 con 0x7ffa28106d80 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.999+0000 7ffa1a7fc700 1 -- 172.21.15.102:0/245945623 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa1c0249d0 con 0x7ffa28106d80 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.999+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2819a8f0 con 0x7ffa28106d80 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.999+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2819d5e0 con 0x7ffa28106d80 2024-03-30T13:52:14.730 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:13.999+0000 7ffa1a7fc700 1 -- 172.21.15.102:0/245945623 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7ffa1c022020 con 0x7ffa28106d80 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.000+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa28063870 con 0x7ffa28106d80 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.000+0000 7ffa1a7fc700 1 --2- 172.21.15.102:0/245945623 >> v2:172.21.15.102:6800/3154847229 conn(0x7ffa14038eb0 0x7ffa1403b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.000+0000 7ffa1a7fc700 1 -- 172.21.15.102:0/245945623 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7ffa1c04d400 con 0x7ffa28106d80 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.000+0000 7ffa2cc2f700 1 --2- 172.21.15.102:0/245945623 >> v2:172.21.15.102:6800/3154847229 conn(0x7ffa14038eb0 0x7ffa1403b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.000+0000 7ffa2cc2f700 1 --2- 172.21.15.102:0/245945623 >> v2:172.21.15.102:6800/3154847229 conn(0x7ffa14038eb0 0x7ffa1403b370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa24006fd0 tx=0x7ffa24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.005+0000 7ffa1a7fc700 1 -- 172.21.15.102:0/245945623 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffa1c02d4a0 con 0x7ffa28106d80 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.201+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7ffa2819da30 con 0x7ffa28106d80 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.202+0000 7ffa1a7fc700 1 -- 172.21.15.102:0/245945623 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7ffa1c02d770 con 0x7ffa28106d80 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 >> v2:172.21.15.102:6800/3154847229 conn(0x7ffa14038eb0 msgr2=0x7ffa1403b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 --2- 172.21.15.102:0/245945623 >> v2:172.21.15.102:6800/3154847229 conn(0x7ffa14038eb0 0x7ffa1403b370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa24006fd0 tx=0x7ffa24006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 msgr2=0x7ffa2819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 --2- 172.21.15.102:0/245945623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa2819a120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c011a10 tx=0x7ffa1c011af0 comp rx=0 tx=0).stop 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 shutdown_connections 2024-03-30T13:52:14.731 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 --2- 172.21.15.102:0/245945623 >> v2:172.21.15.102:6800/3154847229 conn(0x7ffa14038eb0 0x7ffa1403b370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:14.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 --2- 172.21.15.102:0/245945623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa28106d80 0x7ffa2819a120 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:14.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 >> 172.21.15.102:0/245945623 conn(0x7ffa281006f0 msgr2=0x7ffa281010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:14.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 shutdown_connections 2024-03-30T13:52:14.732 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:14.204+0000 7ffa2f694700 1 -- 172.21.15.102:0/245945623 wait complete. 2024-03-30T13:52:15.976 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.114+0000 7fa29cff3700 1 Processor -- start 2024-03-30T13:52:15.976 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.114+0000 7fa29cff3700 1 -- start start 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.114+0000 7fa29cff3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa298109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.114+0000 7fa29cff3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa298079700 con 0x7fa298106d40 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa298109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa298109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51664/0 (socket says 172.21.15.102:51664) 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29659c700 1 -- 172.21.15.102:0/2361970434 learned_addr learned my addr 172.21.15.102:0/2361970434 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29659c700 1 -- 172.21.15.102:0/2361970434 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa298109670 con 0x7fa298106d40 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29659c700 1 --2- 172.21.15.102:0/2361970434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa298109130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2880098a0 tx=0x7fa288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9fc0066f7394aa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29559a700 1 -- 172.21.15.102:0/2361970434 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa288004160 con 0x7fa298106d40 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29559a700 1 -- 172.21.15.102:0/2361970434 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fa288004300 con 0x7fa298106d40 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.115+0000 7fa29559a700 1 -- 172.21.15.102:0/2361970434 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa288011a00 con 0x7fa298106d40 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.116+0000 7fa29cff3700 1 -- 172.21.15.102:0/2361970434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 msgr2=0x7fa298109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.116+0000 7fa29cff3700 1 --2- 172.21.15.102:0/2361970434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa298109130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2880098a0 tx=0x7fa288009bb0 comp rx=0 tx=0).stop 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.116+0000 7fa29cff3700 1 -- 172.21.15.102:0/2361970434 shutdown_connections 2024-03-30T13:52:15.977 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.116+0000 7fa29cff3700 1 --2- 172.21.15.102:0/2361970434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa298109130 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.116+0000 7fa29cff3700 1 -- 172.21.15.102:0/2361970434 >> 172.21.15.102:0/2361970434 conn(0x7fa298100790 msgr2=0x7fa298102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.116+0000 7fa29cff3700 1 -- 172.21.15.102:0/2361970434 shutdown_connections 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.116+0000 7fa29cff3700 1 -- 172.21.15.102:0/2361970434 wait complete. 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29cff3700 1 Processor -- start 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29cff3700 1 -- start start 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29cff3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa29819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29cff3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa29810a620 con 0x7fa298106d40 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa29819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa29819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51672/0 (socket says 172.21.15.102:51672) 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29659c700 1 -- 172.21.15.102:0/1091942198 learned_addr learned my addr 172.21.15.102:0/1091942198 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29659c700 1 -- 172.21.15.102:0/1091942198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa29819a690 con 0x7fa298106d40 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.117+0000 7fa29659c700 1 --2- 172.21.15.102:0/1091942198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa29819a150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa288009870 tx=0x7fa288011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.118+0000 7fa2877fe700 1 -- 172.21.15.102:0/1091942198 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa288003af0 con 0x7fa298106d40 2024-03-30T13:52:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.118+0000 7fa2877fe700 1 -- 172.21.15.102:0/1091942198 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fa28801a8e0 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.118+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa29819a920 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.118+0000 7fa2877fe700 1 -- 172.21.15.102:0/1091942198 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2880248d0 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.118+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa29819d610 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.118+0000 7fa2877fe700 1 -- 172.21.15.102:0/1091942198 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7fa288022020 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.119+0000 7fa2877fe700 1 --2- 172.21.15.102:0/1091942198 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa2800390a0 0x7fa28003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.119+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa298063870 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.119+0000 7fa2877fe700 1 -- 172.21.15.102:0/1091942198 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7fa28804d3e0 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.119+0000 7fa295d9b700 1 --2- 172.21.15.102:0/1091942198 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa2800390a0 0x7fa28003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.119+0000 7fa295d9b700 1 --2- 172.21.15.102:0/1091942198 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa2800390a0 0x7fa28003b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa28c006fd0 tx=0x7fa28c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.124+0000 7fa2877fe700 1 -- 172.21.15.102:0/1091942198 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa28800b3e0 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.423+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fa29819d9f0 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.425+0000 7fa2877fe700 1 -- 172.21.15.102:0/1091942198 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fa2880172b0 con 0x7fa298106d40 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa2800390a0 msgr2=0x7fa28003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 --2- 172.21.15.102:0/1091942198 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa2800390a0 0x7fa28003b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa28c006fd0 tx=0x7fa28c006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 msgr2=0x7fa29819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 --2- 172.21.15.102:0/1091942198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa29819a150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa288009870 tx=0x7fa288011ac0 comp rx=0 tx=0).stop 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 shutdown_connections 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 --2- 172.21.15.102:0/1091942198 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa2800390a0 0x7fa28003b560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 --2- 172.21.15.102:0/1091942198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa298106d40 0x7fa29819a150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 >> 172.21.15.102:0/1091942198 conn(0x7fa298100790 msgr2=0x7fa298101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 shutdown_connections 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-03-30T13:52:15.427+0000 7fa29cff3700 1 -- 172.21.15.102:0/1091942198 wait complete. 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /sbin/cephadm shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout:Or, if you are only running a single cluster on this host: 2024-03-30T13:52:15.980 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /sbin/cephadm shell 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: ceph telemetry on 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout:For more information see: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:15.981 INFO:teuthology.orchestra.run.smithi102.stdout:Bootstrap complete. 2024-03-30T13:52:16.016 INFO:tasks.cephadm:Fetching config... 2024-03-30T13:52:16.017 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:52:16.017 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-30T13:52:16.035 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-30T13:52:16.036 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:52:16.036 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-30T13:52:16.091 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-30T13:52:16.091 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:52:16.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-03-30T13:52:16.158 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-30T13:52:16.158 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:52:16.159 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-30T13:52:16.213 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-30T13:52:16.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3pZrsdGebtcUxK8YVy9J+olJ0RyIk4DwBcY1cfbVlg9GiAH1nJ8LexQpPrAauKowJhW0zkmql4PStok7hzOIJkowepHEO3IYKW7Sif7RufWX3RgksrW78efYKQDj1weBU/adup8yIO6XjvtYhM6oqNJCIAtYxljywAS07JXH0kfgGjQm6/V1lVEma/0aW+rGcjQo48MR7vP9ojtGlXJcgR7FJEKlMf6UQkSf3ga9kIP7ejtrRqKLKowqaOMpvl5k1dbIq/rMe/u0Z7larV77IpcuRUwbmuV0X+cL63/fSBFRi+Sx0wdUDd4Vp1nlTHvSyBQS9Ay2y4MinmzDu7XRMKrC5JRyGhiFoHdjhkq0iLOtliZj82VP1d6Utx825lj4wzlJOGT3r3uDbJpwzRdCC94tMlyIjarBEhJL26PUVi7p5htEbIKLAqdenlvmz4wBkNNj62TND2tH4t8Mx/NnkDVyPI7v9U3lid7DZHRuhAJ5T7PMLupOmd5ynSSQKfVU= ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T13:52:16.294 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3pZrsdGebtcUxK8YVy9J+olJ0RyIk4DwBcY1cfbVlg9GiAH1nJ8LexQpPrAauKowJhW0zkmql4PStok7hzOIJkowepHEO3IYKW7Sif7RufWX3RgksrW78efYKQDj1weBU/adup8yIO6XjvtYhM6oqNJCIAtYxljywAS07JXH0kfgGjQm6/V1lVEma/0aW+rGcjQo48MR7vP9ojtGlXJcgR7FJEKlMf6UQkSf3ga9kIP7ejtrRqKLKowqaOMpvl5k1dbIq/rMe/u0Z7larV77IpcuRUwbmuV0X+cL63/fSBFRi+Sx0wdUDd4Vp1nlTHvSyBQS9Ay2y4MinmzDu7XRMKrC5JRyGhiFoHdjhkq0iLOtliZj82VP1d6Utx825lj4wzlJOGT3r3uDbJpwzRdCC94tMlyIjarBEhJL26PUVi7p5htEbIKLAqdenlvmz4wBkNNj62TND2tH4t8Mx/NnkDVyPI7v9U3lid7DZHRuhAJ5T7PMLupOmd5ynSSQKfVU= ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:16.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3pZrsdGebtcUxK8YVy9J+olJ0RyIk4DwBcY1cfbVlg9GiAH1nJ8LexQpPrAauKowJhW0zkmql4PStok7hzOIJkowepHEO3IYKW7Sif7RufWX3RgksrW78efYKQDj1weBU/adup8yIO6XjvtYhM6oqNJCIAtYxljywAS07JXH0kfgGjQm6/V1lVEma/0aW+rGcjQo48MR7vP9ojtGlXJcgR7FJEKlMf6UQkSf3ga9kIP7ejtrRqKLKowqaOMpvl5k1dbIq/rMe/u0Z7larV77IpcuRUwbmuV0X+cL63/fSBFRi+Sx0wdUDd4Vp1nlTHvSyBQS9Ay2y4MinmzDu7XRMKrC5JRyGhiFoHdjhkq0iLOtliZj82VP1d6Utx825lj4wzlJOGT3r3uDbJpwzRdCC94tMlyIjarBEhJL26PUVi7p5htEbIKLAqdenlvmz4wBkNNj62TND2tH4t8Mx/NnkDVyPI7v9U3lid7DZHRuhAJ5T7PMLupOmd5ynSSQKfVU= ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T13:52:16.353 INFO:teuthology.orchestra.run.smithi138.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3pZrsdGebtcUxK8YVy9J+olJ0RyIk4DwBcY1cfbVlg9GiAH1nJ8LexQpPrAauKowJhW0zkmql4PStok7hzOIJkowepHEO3IYKW7Sif7RufWX3RgksrW78efYKQDj1weBU/adup8yIO6XjvtYhM6oqNJCIAtYxljywAS07JXH0kfgGjQm6/V1lVEma/0aW+rGcjQo48MR7vP9ojtGlXJcgR7FJEKlMf6UQkSf3ga9kIP7ejtrRqKLKowqaOMpvl5k1dbIq/rMe/u0Z7larV77IpcuRUwbmuV0X+cL63/fSBFRi+Sx0wdUDd4Vp1nlTHvSyBQS9Ay2y4MinmzDu7XRMKrC5JRyGhiFoHdjhkq0iLOtliZj82VP1d6Utx825lj4wzlJOGT3r3uDbJpwzRdCC94tMlyIjarBEhJL26PUVi7p5htEbIKLAqdenlvmz4wBkNNj62TND2tH4t8Mx/NnkDVyPI7v9U3lid7DZHRuhAJ5T7PMLupOmd5ynSSQKfVU= ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:16.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-30T13:52:16.513 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1091942198' entity='client.admin' 2024-03-30T13:52:16.606 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.867+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1545611689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 msgr2=0x7f8bf0104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.867+0000 7f8bf78e0700 1 --2- 172.21.15.102:0/1545611689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0104bc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009910 tx=0x7f8be0009c20 comp rx=0 tx=0).stop 2024-03-30T13:52:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.867+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1545611689 shutdown_connections 2024-03-30T13:52:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.867+0000 7f8bf78e0700 1 --2- 172.21.15.102:0/1545611689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0104bc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.867+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1545611689 >> 172.21.15.102:0/1545611689 conn(0x7f8bf00fc140 msgr2=0x7f8bf00fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1545611689 shutdown_connections 2024-03-30T13:52:18.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1545611689 wait complete. 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf78e0700 1 Processor -- start 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf78e0700 1 -- start start 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf78e0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf78e0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf01073c0 con 0x7f8bf01027d0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf567c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf567c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51690/0 (socket says 172.21.15.102:51690) 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.868+0000 7f8bf567c700 1 -- 172.21.15.102:0/1597822002 learned_addr learned my addr 172.21.15.102:0/1597822002 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.869+0000 7f8bf567c700 1 -- 172.21.15.102:0/1597822002 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf0196ff0 con 0x7f8bf01027d0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.869+0000 7f8bf567c700 1 --2- 172.21.15.102:0/1597822002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0196ab0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8be000b5c0 tx=0x7f8be0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.869+0000 7f8be67fc700 1 -- 172.21.15.102:0/1597822002 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8be0027070 con 0x7f8bf01027d0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.869+0000 7f8be67fc700 1 -- 172.21.15.102:0/1597822002 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f8be000be10 con 0x7f8bf01027d0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.869+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf0197280 con 0x7f8bf01027d0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.869+0000 7f8be67fc700 1 -- 172.21.15.102:0/1597822002 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8be0022da0 con 0x7f8bf01027d0 2024-03-30T13:52:18.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.869+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf0199f70 con 0x7f8bf01027d0 2024-03-30T13:52:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.870+0000 7f8be67fc700 1 -- 172.21.15.102:0/1597822002 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f8be002a070 con 0x7f8bf01027d0 2024-03-30T13:52:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.870+0000 7f8be67fc700 1 --2- 172.21.15.102:0/1597822002 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8bdc038e60 0x7f8bdc03b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:18.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.870+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf0107b40 con 0x7f8bf01027d0 2024-03-30T13:52:18.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.870+0000 7f8be67fc700 1 -- 172.21.15.102:0/1597822002 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7f8be001e030 con 0x7f8bf01027d0 2024-03-30T13:52:18.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.870+0000 7f8bf4e7b700 1 --2- 172.21.15.102:0/1597822002 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8bdc038e60 0x7f8bdc03b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:18.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.870+0000 7f8bf4e7b700 1 --2- 172.21.15.102:0/1597822002 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8bdc038e60 0x7f8bdc03b320 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8bec006fd0 tx=0x7f8bec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:18.875+0000 7f8be67fc700 1 -- 172.21.15.102:0/1597822002 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8be0013070 con 0x7f8bf01027d0 2024-03-30T13:52:19.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.073+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f8bf002db30 con 0x7f8bf01027d0 2024-03-30T13:52:19.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.078+0000 7f8be67fc700 1 -- 172.21.15.102:0/1597822002 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f8be002b570 con 0x7f8bf01027d0 2024-03-30T13:52:19.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8bdc038e60 msgr2=0x7f8bdc03b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:19.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 --2- 172.21.15.102:0/1597822002 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8bdc038e60 0x7f8bdc03b320 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8bec006fd0 tx=0x7f8bec006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 msgr2=0x7f8bf0196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 --2- 172.21.15.102:0/1597822002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0196ab0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8be000b5c0 tx=0x7f8be0005030 comp rx=0 tx=0).stop 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 shutdown_connections 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 --2- 172.21.15.102:0/1597822002 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8bdc038e60 0x7f8bdc03b320 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 --2- 172.21.15.102:0/1597822002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8bf01027d0 0x7f8bf0196ab0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 >> 172.21.15.102:0/1597822002 conn(0x7f8bf00fc140 msgr2=0x7f8bf00fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 shutdown_connections 2024-03-30T13:52:19.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:19.080+0000 7f8bf78e0700 1 -- 172.21.15.102:0/1597822002 wait complete. 2024-03-30T13:52:19.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:19.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:19.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:52:19.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:19.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:19.091 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:19.734 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-30T13:52:19.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-30T13:52:19.967 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:20.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1597822002' entity='client.admin' 2024-03-30T13:52:20.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:20 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:20.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:20 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:21.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 -- 172.21.15.102:0/2859264746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 msgr2=0x7efee0102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:21.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 --2- 172.21.15.102:0/2859264746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0102090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efed0009910 tx=0x7efed0009c20 comp rx=0 tx=0).stop 2024-03-30T13:52:21.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 -- 172.21.15.102:0/2859264746 shutdown_connections 2024-03-30T13:52:21.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 --2- 172.21.15.102:0/2859264746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0102090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:21.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 -- 172.21.15.102:0/2859264746 >> 172.21.15.102:0/2859264746 conn(0x7efee00fd230 msgr2=0x7efee00ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:21.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 -- 172.21.15.102:0/2859264746 shutdown_connections 2024-03-30T13:52:21.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 -- 172.21.15.102:0/2859264746 wait complete. 2024-03-30T13:52:21.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 Processor -- start 2024-03-30T13:52:21.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.464+0000 7efee63ac700 1 -- start start 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efee63ac700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0075f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efee63ac700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efee01073b0 con 0x7efee0101c70 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0075f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0075f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33150/0 (socket says 172.21.15.102:33150) 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedffff700 1 -- 172.21.15.102:0/2380695885 learned_addr learned my addr 172.21.15.102:0/2380695885 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedffff700 1 -- 172.21.15.102:0/2380695885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee0076440 con 0x7efee0101c70 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedffff700 1 --2- 172.21.15.102:0/2380695885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0075f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efed000b5c0 tx=0x7efed00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedd7fa700 1 -- 172.21.15.102:0/2380695885 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efed0027070 con 0x7efee0101c70 2024-03-30T13:52:21.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedd7fa700 1 -- 172.21.15.102:0/2380695885 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7efed000be10 con 0x7efee0101c70 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee0074630 con 0x7efee0101c70 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efedd7fa700 1 -- 172.21.15.102:0/2380695885 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efed0022e40 con 0x7efee0101c70 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.465+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee0074b70 con 0x7efee0101c70 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.466+0000 7efedd7fa700 1 -- 172.21.15.102:0/2380695885 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7efed002a070 con 0x7efee0101c70 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.466+0000 7efedd7fa700 1 --2- 172.21.15.102:0/2380695885 >> v2:172.21.15.102:6800/3154847229 conn(0x7efec8038e60 0x7efec803b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.466+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee0107b30 con 0x7efee0101c70 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.466+0000 7efedd7fa700 1 -- 172.21.15.102:0/2380695885 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7efed001e030 con 0x7efee0101c70 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.466+0000 7efedf7fe700 1 --2- 172.21.15.102:0/2380695885 >> v2:172.21.15.102:6800/3154847229 conn(0x7efec8038e60 0x7efec803b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.467+0000 7efedf7fe700 1 --2- 172.21.15.102:0/2380695885 >> v2:172.21.15.102:6800/3154847229 conn(0x7efec8038e60 0x7efec803b320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efed4006fd0 tx=0x7efed4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.471+0000 7efedd7fa700 1 -- 172.21.15.102:0/2380695885 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efed0013070 con 0x7efee0101c70 2024-03-30T13:52:21.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.669+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7efee0074eb0 con 0x7efec8038e60 2024-03-30T13:52:21.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.673+0000 7efedd7fa700 1 -- 172.21.15.102:0/2380695885 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efee0074eb0 con 0x7efec8038e60 2024-03-30T13:52:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 >> v2:172.21.15.102:6800/3154847229 conn(0x7efec8038e60 msgr2=0x7efec803b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 --2- 172.21.15.102:0/2380695885 >> v2:172.21.15.102:6800/3154847229 conn(0x7efec8038e60 0x7efec803b320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efed4006fd0 tx=0x7efed4006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 msgr2=0x7efee0075f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 --2- 172.21.15.102:0/2380695885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0075f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efed000b5c0 tx=0x7efed00046b0 comp rx=0 tx=0).stop 2024-03-30T13:52:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 shutdown_connections 2024-03-30T13:52:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 --2- 172.21.15.102:0/2380695885 >> v2:172.21.15.102:6800/3154847229 conn(0x7efec8038e60 0x7efec803b320 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 --2- 172.21.15.102:0/2380695885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efee0101c70 0x7efee0075f00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:21.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 >> 172.21.15.102:0/2380695885 conn(0x7efee00fd230 msgr2=0x7efee00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:21.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 shutdown_connections 2024-03-30T13:52:21.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:21.675+0000 7efee63ac700 1 -- 172.21.15.102:0/2380695885 wait complete. 2024-03-30T13:52:22.417 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi138 2024-03-30T13:52:22.417 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:52:22.417 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.conf 2024-03-30T13:52:22.435 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:52:22.435 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:52:22.492 INFO:tasks.cephadm:Adding host smithi138 to orchestrator... 2024-03-30T13:52:22.493 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi138 2024-03-30T13:52:22.731 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:22.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:52:22.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:22.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:22.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:22.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:22.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-30T13:52:22.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:22.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:52:22.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:22.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:22.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:23.770 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:23 smithi102 ceph-mon[29392]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:52:24.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 -- 172.21.15.102:0/39115966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 msgr2=0x7f1644106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 --2- 172.21.15.102:0/39115966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644106fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f163c00b0a0 tx=0x7f163c00b3b0 comp rx=0 tx=0).stop 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 -- 172.21.15.102:0/39115966 shutdown_connections 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 --2- 172.21.15.102:0/39115966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644106fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 -- 172.21.15.102:0/39115966 >> 172.21.15.102:0/39115966 conn(0x7f16440fc670 msgr2=0x7f16440fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 -- 172.21.15.102:0/39115966 shutdown_connections 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 -- 172.21.15.102:0/39115966 wait complete. 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 Processor -- start 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.243+0000 7f164b379700 1 -- start start 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f164b379700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f164b379700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f164410a860 con 0x7f1644106bf0 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f1649115700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f1649115700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33166/0 (socket says 172.21.15.102:33166) 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f1649115700 1 -- 172.21.15.102:0/2616825342 learned_addr learned my addr 172.21.15.102:0/2616825342 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f1649115700 1 -- 172.21.15.102:0/2616825342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16441981d0 con 0x7f1644106bf0 2024-03-30T13:52:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f1649115700 1 --2- 172.21.15.102:0/2616825342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644197c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f163c0062a0 tx=0x7f163c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f163a7fc700 1 -- 172.21.15.102:0/2616825342 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f163c017040 con 0x7f1644106bf0 2024-03-30T13:52:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f163a7fc700 1 -- 172.21.15.102:0/2616825342 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f163c022070 con 0x7f1644106bf0 2024-03-30T13:52:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1644198460 con 0x7f1644106bf0 2024-03-30T13:52:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f163a7fc700 1 -- 172.21.15.102:0/2616825342 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f163c004900 con 0x7f1644106bf0 2024-03-30T13:52:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.244+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16441988c0 con 0x7f1644106bf0 2024-03-30T13:52:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.245+0000 7f163a7fc700 1 -- 172.21.15.102:0/2616825342 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f163c027460 con 0x7f1644106bf0 2024-03-30T13:52:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.245+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16441021f0 con 0x7f1644106bf0 2024-03-30T13:52:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.245+0000 7f163a7fc700 1 --2- 172.21.15.102:0/2616825342 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1630038eb0 0x7f163003b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.245+0000 7f163a7fc700 1 -- 172.21.15.102:0/2616825342 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7f163c00f050 con 0x7f1644106bf0 2024-03-30T13:52:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.245+0000 7f1648914700 1 --2- 172.21.15.102:0/2616825342 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1630038eb0 0x7f163003b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.246+0000 7f1648914700 1 --2- 172.21.15.102:0/2616825342 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1630038eb0 0x7f163003b370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1634006fd0 tx=0x7f1634006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:24.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.250+0000 7f163a7fc700 1 -- 172.21.15.102:0/2616825342 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f163c027850 con 0x7f1644106bf0 2024-03-30T13:52:24.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:24.449+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}) -- 0x7f1644063870 con 0x7f1630038eb0 2024-03-30T13:52:25.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:25 smithi102 ceph-mon[29392]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:52:25.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:25 smithi102 ceph-mon[29392]: Deploying cephadm binary to smithi138 2024-03-30T13:52:28.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:27 smithi102 ceph-mon[29392]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:28.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.689+0000 7f163a7fc700 1 -- 172.21.15.102:0/2616825342 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f1644063870 con 0x7f1630038eb0 2024-03-30T13:52:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi138' with addr '172.21.15.138' 2024-03-30T13:52:28.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1630038eb0 msgr2=0x7f163003b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:28.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 --2- 172.21.15.102:0/2616825342 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1630038eb0 0x7f163003b370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1634006fd0 tx=0x7f1634006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:28.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 msgr2=0x7f1644197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 --2- 172.21.15.102:0/2616825342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644197c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f163c0062a0 tx=0x7f163c00bd40 comp rx=0 tx=0).stop 2024-03-30T13:52:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 shutdown_connections 2024-03-30T13:52:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 --2- 172.21.15.102:0/2616825342 >> v2:172.21.15.102:6800/3154847229 conn(0x7f1630038eb0 0x7f163003b370 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 --2- 172.21.15.102:0/2616825342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644106bf0 0x7f1644197c90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.691+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 >> 172.21.15.102:0/2616825342 conn(0x7f16440fc670 msgr2=0x7f16440fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.692+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 shutdown_connections 2024-03-30T13:52:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:28.692+0000 7f164b379700 1 -- 172.21.15.102:0/2616825342 wait complete. 2024-03-30T13:52:29.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T13:52:29.652 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:29.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:29.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:29 smithi102 ceph-mon[29392]: Added host smithi138 2024-03-30T13:52:29.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:29.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:29 smithi102 ceph-mon[29392]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:29.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:30.973 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:31.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 -- 172.21.15.102:0/2666738491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 msgr2=0x7f86ac0749c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:31.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 --2- 172.21.15.102:0/2666738491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac0749c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009910 tx=0x7f86a8009c20 comp rx=0 tx=0).stop 2024-03-30T13:52:31.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 -- 172.21.15.102:0/2666738491 shutdown_connections 2024-03-30T13:52:31.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 --2- 172.21.15.102:0/2666738491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac0749c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:31.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 -- 172.21.15.102:0/2666738491 >> 172.21.15.102:0/2666738491 conn(0x7f86ac0fd730 msgr2=0x7f86ac0ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 -- 172.21.15.102:0/2666738491 shutdown_connections 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 -- 172.21.15.102:0/2666738491 wait complete. 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 Processor -- start 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.204+0000 7f86b2faf700 1 -- start start 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b2faf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac19d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b2faf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f86ac111f80 con 0x7f86ac0745e0 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b0d4b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac19d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b0d4b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac19d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35022/0 (socket says 172.21.15.102:35022) 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b0d4b700 1 -- 172.21.15.102:0/2460798292 learned_addr learned my addr 172.21.15.102:0/2460798292 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b0d4b700 1 -- 172.21.15.102:0/2460798292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86ac19da20 con 0x7f86ac0745e0 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b0d4b700 1 --2- 172.21.15.102:0/2460798292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac19d4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f86a8004ed0 tx=0x7f86a8004fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86a1ffb700 1 -- 172.21.15.102:0/2460798292 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86a8026070 con 0x7f86ac0745e0 2024-03-30T13:52:31.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86a1ffb700 1 -- 172.21.15.102:0/2460798292 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f86a800b750 con 0x7f86ac0745e0 2024-03-30T13:52:31.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86ac19dcb0 con 0x7f86ac0745e0 2024-03-30T13:52:31.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.205+0000 7f86a1ffb700 1 -- 172.21.15.102:0/2460798292 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86a80039a0 con 0x7f86ac0745e0 2024-03-30T13:52:31.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.206+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86ac19e110 con 0x7f86ac0745e0 2024-03-30T13:52:31.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.206+0000 7f86a1ffb700 1 -- 172.21.15.102:0/2460798292 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f86a8029070 con 0x7f86ac0745e0 2024-03-30T13:52:31.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.207+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86ac075d20 con 0x7f86ac0745e0 2024-03-30T13:52:31.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.207+0000 7f86a1ffb700 1 --2- 172.21.15.102:0/2460798292 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8694038eb0 0x7f869403b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:31.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.207+0000 7f86a1ffb700 1 -- 172.21.15.102:0/2460798292 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7f86a801e070 con 0x7f86ac0745e0 2024-03-30T13:52:31.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.207+0000 7f86a3fff700 1 --2- 172.21.15.102:0/2460798292 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8694038eb0 0x7f869403b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:31.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.207+0000 7f86a3fff700 1 --2- 172.21.15.102:0/2460798292 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8694038eb0 0x7f869403b370 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f869c006fd0 tx=0x7f869c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:31.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.212+0000 7f86a1ffb700 1 -- 172.21.15.102:0/2460798292 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f86a8013070 con 0x7f86ac0745e0 2024-03-30T13:52:31.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.411+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f86ac19e840 con 0x7f8694038eb0 2024-03-30T13:52:31.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.412+0000 7f86a1ffb700 1 -- 172.21.15.102:0/2460798292 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f86ac19e840 con 0x7f8694038eb0 2024-03-30T13:52:31.412 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:52:31.412 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.138", "hostname": "smithi138", "labels": [], "status": ""}] 2024-03-30T13:52:31.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8694038eb0 msgr2=0x7f869403b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:31.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 --2- 172.21.15.102:0/2460798292 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8694038eb0 0x7f869403b370 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f869c006fd0 tx=0x7f869c006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:31.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 msgr2=0x7f86ac19d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:31.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 --2- 172.21.15.102:0/2460798292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac19d4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f86a8004ed0 tx=0x7f86a8004fb0 comp rx=0 tx=0).stop 2024-03-30T13:52:31.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 shutdown_connections 2024-03-30T13:52:31.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 --2- 172.21.15.102:0/2460798292 >> v2:172.21.15.102:6800/3154847229 conn(0x7f8694038eb0 0x7f869403b370 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:31.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 --2- 172.21.15.102:0/2460798292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86ac0745e0 0x7f86ac19d4e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:31.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 >> 172.21.15.102:0/2460798292 conn(0x7f86ac0fd730 msgr2=0x7f86ac0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:31.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 shutdown_connections 2024-03-30T13:52:31.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:31.414+0000 7f86b2faf700 1 -- 172.21.15.102:0/2460798292 wait complete. 2024-03-30T13:52:31.998 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:31 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:31.999 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:31 smithi102 ceph-mon[29392]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:32.028 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-30T13:52:32.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-03-30T13:52:32.269 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:33.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:32 smithi102 ceph-mon[29392]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 -- 172.21.15.102:0/3006335611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 msgr2=0x7f1934106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 --2- 172.21.15.102:0/3006335611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f1934106fd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1924009910 tx=0x7f1924009c20 comp rx=0 tx=0).stop 2024-03-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 -- 172.21.15.102:0/3006335611 shutdown_connections 2024-03-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 --2- 172.21.15.102:0/3006335611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f1934106fd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 -- 172.21.15.102:0/3006335611 >> 172.21.15.102:0/3006335611 conn(0x7f19340fc670 msgr2=0x7f19340fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 -- 172.21.15.102:0/3006335611 shutdown_connections 2024-03-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 -- 172.21.15.102:0/3006335611 wait complete. 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 Processor -- start 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.843+0000 7f193a368700 1 -- start start 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f193a368700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f193419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f193a368700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f193406afd0 con 0x7f1934106bf0 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f1933fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f193419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f1933fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f193419a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35036/0 (socket says 172.21.15.102:35036) 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f1933fff700 1 -- 172.21.15.102:0/3001285721 learned_addr learned my addr 172.21.15.102:0/3001285721 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f1933fff700 1 -- 172.21.15.102:0/3001285721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193419ae60 con 0x7f1934106bf0 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f1933fff700 1 --2- 172.21.15.102:0/3001285721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f193419a920 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1924004f80 tx=0x7f1924005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.844+0000 7f19317fa700 1 -- 172.21.15.102:0/3001285721 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1924026070 con 0x7f1934106bf0 2024-03-30T13:52:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.845+0000 7f19317fa700 1 -- 172.21.15.102:0/3001285721 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f192400bc00 con 0x7f1934106bf0 2024-03-30T13:52:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.845+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193419b0f0 con 0x7f1934106bf0 2024-03-30T13:52:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.845+0000 7f19317fa700 1 -- 172.21.15.102:0/3001285721 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1924003990 con 0x7f1934106bf0 2024-03-30T13:52:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.845+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1934194ae0 con 0x7f1934106bf0 2024-03-30T13:52:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.845+0000 7f19317fa700 1 -- 172.21.15.102:0/3001285721 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f1924029070 con 0x7f1934106bf0 2024-03-30T13:52:33.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.846+0000 7f19317fa700 1 --2- 172.21.15.102:0/3001285721 >> v2:172.21.15.102:6800/3154847229 conn(0x7f191c03b0b0 0x7f191c03d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.846+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19341021f0 con 0x7f1934106bf0 2024-03-30T13:52:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.846+0000 7f19317fa700 1 -- 172.21.15.102:0/3001285721 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (secure 0 0 0) 0x7f192401e070 con 0x7f1934106bf0 2024-03-30T13:52:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.846+0000 7f19337fe700 1 --2- 172.21.15.102:0/3001285721 >> v2:172.21.15.102:6800/3154847229 conn(0x7f191c03b0b0 0x7f191c03d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:33.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.846+0000 7f19337fe700 1 --2- 172.21.15.102:0/3001285721 >> v2:172.21.15.102:6800/3154847229 conn(0x7f191c03b0b0 0x7f191c03d570 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1928006fd0 tx=0x7f1928006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:33.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:33.851+0000 7f19317fa700 1 -- 172.21.15.102:0/3001285721 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1924013070 con 0x7f1934106bf0 2024-03-30T13:52:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.053+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f1934063870 con 0x7f1934106bf0 2024-03-30T13:52:34.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:33 smithi102 ceph-mon[29392]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.912+0000 7f19317fa700 1 -- 172.21.15.102:0/3001285721 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f1924030020 con 0x7f1934106bf0 2024-03-30T13:52:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 >> v2:172.21.15.102:6800/3154847229 conn(0x7f191c03b0b0 msgr2=0x7f191c03d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 --2- 172.21.15.102:0/3001285721 >> v2:172.21.15.102:6800/3154847229 conn(0x7f191c03b0b0 0x7f191c03d570 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1928006fd0 tx=0x7f1928006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 msgr2=0x7f193419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 --2- 172.21.15.102:0/3001285721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f193419a920 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1924004f80 tx=0x7f1924005060 comp rx=0 tx=0).stop 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 shutdown_connections 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 --2- 172.21.15.102:0/3001285721 >> v2:172.21.15.102:6800/3154847229 conn(0x7f191c03b0b0 0x7f191c03d570 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 --2- 172.21.15.102:0/3001285721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1934106bf0 0x7f193419a920 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 >> 172.21.15.102:0/3001285721 conn(0x7f19340fc670 msgr2=0x7f19340fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 shutdown_connections 2024-03-30T13:52:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:52:34.915+0000 7f193a368700 1 -- 172.21.15.102:0/3001285721 wait complete. 2024-03-30T13:52:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-03-30T13:52:35.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3001285721' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T13:52:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:35.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:35.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:35.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:34 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:35.596 INFO:tasks.cephadm:Adding mon.a on smithi102 2024-03-30T13:52:35.597 INFO:tasks.cephadm:Adding mon.c on smithi102 2024-03-30T13:52:35.597 INFO:tasks.cephadm:Adding mon.b on smithi138 2024-03-30T13:52:35.597 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi138:172.21.15.138=b' 2024-03-30T13:52:35.835 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:36.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:35 smithi102 ceph-mon[29392]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:52:36.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:35 smithi102 ceph-mon[29392]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:52:36.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3001285721' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T13:52:36.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:35 smithi102 ceph-mon[29392]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:52:36.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:35 smithi102 ceph-mon[29392]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:37.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.236+0000 7f651632c700 1 -- 172.21.15.138:0/2887702956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 msgr2=0x7f65101031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:37.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.236+0000 7f651632c700 1 --2- 172.21.15.138:0/2887702956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f65101031c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6500009910 tx=0x7f6500009c20 comp rx=0 tx=0).stop 2024-03-30T13:52:37.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.236+0000 7f651632c700 1 -- 172.21.15.138:0/2887702956 shutdown_connections 2024-03-30T13:52:37.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.236+0000 7f651632c700 1 --2- 172.21.15.138:0/2887702956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f65101031c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:37.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.236+0000 7f651632c700 1 -- 172.21.15.138:0/2887702956 >> 172.21.15.138:0/2887702956 conn(0x7f65100fc650 msgr2=0x7f65100fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:37.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.236+0000 7f651632c700 1 -- 172.21.15.138:0/2887702956 shutdown_connections 2024-03-30T13:52:37.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.236+0000 7f651632c700 1 -- 172.21.15.138:0/2887702956 wait complete. 2024-03-30T13:52:37.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.237+0000 7f651632c700 1 Processor -- start 2024-03-30T13:52:37.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.237+0000 7f651632c700 1 -- start start 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.237+0000 7f651632c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f6510196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.237+0000 7f651632c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f651010b960 con 0x7f6510102de0 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.237+0000 7f650ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f6510196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f650ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f6510196fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.138:53528/0 (socket says 172.21.15.138:53528) 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f650ffff700 1 -- 172.21.15.138:0/869013607 learned_addr learned my addr 172.21.15.138:0/869013607 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f650ffff700 1 -- 172.21.15.138:0/869013607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65101974f0 con 0x7f6510102de0 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f650ffff700 1 --2- 172.21.15.138:0/869013607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f6510196fb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6500004ed0 tx=0x7f6500004fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:37.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f650d7fa700 1 -- 172.21.15.138:0/869013607 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6500026070 con 0x7f6510102de0 2024-03-30T13:52:37.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6510197780 con 0x7f6510102de0 2024-03-30T13:52:37.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f650d7fa700 1 -- 172.21.15.138:0/869013607 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f650000b750 con 0x7f6510102de0 2024-03-30T13:52:37.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f650d7fa700 1 -- 172.21.15.138:0/869013607 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65000039a0 con 0x7f6510102de0 2024-03-30T13:52:37.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.238+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6510197be0 con 0x7f6510102de0 2024-03-30T13:52:37.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.239+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651006a930 con 0x7f6510102de0 2024-03-30T13:52:37.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.239+0000 7f650d7fa700 1 -- 172.21.15.138:0/869013607 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f6500029070 con 0x7f6510102de0 2024-03-30T13:52:37.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.239+0000 7f650d7fa700 1 --2- 172.21.15.138:0/869013607 >> v2:172.21.15.102:6800/3154847229 conn(0x7f64f8038e60 0x7f64f803b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:37.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.239+0000 7f650d7fa700 1 -- 172.21.15.138:0/869013607 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (secure 0 0 0) 0x7f650001e070 con 0x7f6510102de0 2024-03-30T13:52:37.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.240+0000 7f650f7fe700 1 --2- 172.21.15.138:0/869013607 >> v2:172.21.15.102:6800/3154847229 conn(0x7f64f8038e60 0x7f64f803b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:37.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.240+0000 7f650f7fe700 1 --2- 172.21.15.138:0/869013607 >> v2:172.21.15.102:6800/3154847229 conn(0x7f64f8038e60 0x7f64f803b320 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6504006fd0 tx=0x7f6504006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:37.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.245+0000 7f650d7fa700 1 -- 172.21.15.138:0/869013607 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6500013070 con 0x7f6510102de0 2024-03-30T13:52:37.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.451+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi138:172.21.15.138=b", "target": ["mon-mgr", ""]}) -- 0x7f6510198270 con 0x7f64f8038e60 2024-03-30T13:52:37.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.460+0000 7f650d7fa700 1 -- 172.21.15.138:0/869013607 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6510198270 con 0x7f64f8038e60 2024-03-30T13:52:37.460 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mon update... 2024-03-30T13:52:37.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 >> v2:172.21.15.102:6800/3154847229 conn(0x7f64f8038e60 msgr2=0x7f64f803b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:37.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 --2- 172.21.15.138:0/869013607 >> v2:172.21.15.102:6800/3154847229 conn(0x7f64f8038e60 0x7f64f803b320 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6504006fd0 tx=0x7f6504006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:37.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 msgr2=0x7f6510196fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:37.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 --2- 172.21.15.138:0/869013607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f6510196fb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6500004ed0 tx=0x7f6500004fb0 comp rx=0 tx=0).stop 2024-03-30T13:52:37.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 shutdown_connections 2024-03-30T13:52:37.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 --2- 172.21.15.138:0/869013607 >> v2:172.21.15.102:6800/3154847229 conn(0x7f64f8038e60 0x7f64f803b320 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:37.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 --2- 172.21.15.138:0/869013607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6510102de0 0x7f6510196fb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:37.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.462+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 >> 172.21.15.138:0/869013607 conn(0x7f65100fc650 msgr2=0x7f65100fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:37.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.463+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 shutdown_connections 2024-03-30T13:52:37.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:37.463+0000 7f651632c700 1 -- 172.21.15.138:0/869013607 wait complete. 2024-03-30T13:52:38.062 DEBUG:teuthology.orchestra.run.smithi102:mon.c> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.c.service 2024-03-30T13:52:38.064 DEBUG:teuthology.orchestra.run.smithi138:mon.b> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.b.service 2024-03-30T13:52:38.067 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T13:52:38.067 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:52:38.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:38 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:39.200 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:39 smithi102 ceph-mon[29392]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi138:172.21.15.138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:52:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:39 smithi102 ceph-mon[29392]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi138:172.21.15.138=b;count:3 2024-03-30T13:52:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:39 smithi102 ceph-mon[29392]: Deploying daemon mon.b on smithi138 2024-03-30T13:52:40.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:40 smithi102 ceph-mon[29392]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:41.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.873+0000 7f907689c700 1 -- 172.21.15.138:0/344709101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 msgr2=0x7f9070100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:41.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.873+0000 7f907689c700 1 --2- 172.21.15.138:0/344709101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070100790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9060009910 tx=0x7f9060009c20 comp rx=0 tx=0).stop 2024-03-30T13:52:41.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.873+0000 7f907689c700 1 -- 172.21.15.138:0/344709101 shutdown_connections 2024-03-30T13:52:41.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.873+0000 7f907689c700 1 --2- 172.21.15.138:0/344709101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070100790 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.873+0000 7f907689c700 1 -- 172.21.15.138:0/344709101 >> 172.21.15.138:0/344709101 conn(0x7f90700fbd70 msgr2=0x7f90700fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.874+0000 7f907689c700 1 -- 172.21.15.138:0/344709101 shutdown_connections 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.874+0000 7f907689c700 1 -- 172.21.15.138:0/344709101 wait complete. 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.874+0000 7f907689c700 1 Processor -- start 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.874+0000 7f907689c700 1 -- start start 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.875+0000 7f907689c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.875+0000 7f907689c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f907010b8f0 con 0x7f90701003b0 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.875+0000 7f906ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.875+0000 7f906ffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.138:53544/0 (socket says 172.21.15.138:53544) 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.875+0000 7f906ffff700 1 -- 172.21.15.138:0/3369530551 learned_addr learned my addr 172.21.15.138:0/3369530551 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.875+0000 7f906ffff700 1 -- 172.21.15.138:0/3369530551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90701973d0 con 0x7f90701003b0 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.875+0000 7f906ffff700 1 --2- 172.21.15.138:0/3369530551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070196e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9060004f80 tx=0x7f9060005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.876+0000 7f906d7fa700 1 -- 172.21.15.138:0/3369530551 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9060026070 con 0x7f90701003b0 2024-03-30T13:52:41.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.876+0000 7f906d7fa700 1 -- 172.21.15.138:0/3369530551 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f906000bba0 con 0x7f90701003b0 2024-03-30T13:52:41.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.876+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90701976c0 con 0x7f90701003b0 2024-03-30T13:52:41.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.876+0000 7f906d7fa700 1 -- 172.21.15.138:0/3369530551 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9060003930 con 0x7f90701003b0 2024-03-30T13:52:41.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.876+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f907019b760 con 0x7f90701003b0 2024-03-30T13:52:41.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.877+0000 7f906d7fa700 1 -- 172.21.15.138:0/3369530551 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f9060029070 con 0x7f90701003b0 2024-03-30T13:52:41.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.877+0000 7f906d7fa700 1 --2- 172.21.15.138:0/3369530551 >> v2:172.21.15.102:6800/3154847229 conn(0x7f905c038e60 0x7f905c03b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:41.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.877+0000 7f906d7fa700 1 -- 172.21.15.138:0/3369530551 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (secure 0 0 0) 0x7f906001e070 con 0x7f90701003b0 2024-03-30T13:52:41.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.877+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9070103470 con 0x7f90701003b0 2024-03-30T13:52:41.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.877+0000 7f906f7fe700 1 --2- 172.21.15.138:0/3369530551 >> v2:172.21.15.102:6800/3154847229 conn(0x7f905c038e60 0x7f905c03b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:41.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.877+0000 7f906f7fe700 1 --2- 172.21.15.138:0/3369530551 >> v2:172.21.15.102:6800/3154847229 conn(0x7f905c038e60 0x7f905c03b320 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9058006fd0 tx=0x7f9058006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:41.882 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:41.882+0000 7f906d7fa700 1 -- 172.21.15.138:0/3369530551 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9060030080 con 0x7f90701003b0 2024-03-30T13:52:42.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.174+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9070063870 con 0x7f90701003b0 2024-03-30T13:52:42.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.175+0000 7f906d7fa700 1 -- 172.21.15.138:0/3369530551 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f9060013070 con 0x7f90701003b0 2024-03-30T13:52:42.175 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:52:42.175 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":1,"fsid":"69dd7ec8-ee9c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T13:51:09.933456Z","created":"2024-03-30T13:51:09.933456Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T13:52:42.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 >> v2:172.21.15.102:6800/3154847229 conn(0x7f905c038e60 msgr2=0x7f905c03b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:42.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 --2- 172.21.15.138:0/3369530551 >> v2:172.21.15.102:6800/3154847229 conn(0x7f905c038e60 0x7f905c03b320 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9058006fd0 tx=0x7f9058006e40 comp rx=0 tx=0).stop 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 msgr2=0x7f9070196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 --2- 172.21.15.138:0/3369530551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070196e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9060004f80 tx=0x7f9060005060 comp rx=0 tx=0).stop 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 shutdown_connections 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 --2- 172.21.15.138:0/3369530551 >> v2:172.21.15.102:6800/3154847229 conn(0x7f905c038e60 0x7f905c03b320 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 --2- 172.21.15.138:0/3369530551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90701003b0 0x7f9070196e90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 >> 172.21.15.138:0/3369530551 conn(0x7f90700fbd70 msgr2=0x7f9070109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 shutdown_connections 2024-03-30T13:52:42.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:42.177+0000 7f907689c700 1 -- 172.21.15.138:0/3369530551 wait complete. 2024-03-30T13:52:42.178 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 1 2024-03-30T13:52:42.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:42 smithi102 ceph-mon[29392]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:43.843 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T13:52:43.843 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T13:52:44.088 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:47.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: mon.a calling monitor election 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: mon.b calling monitor election 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:47.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: monmap epoch 2 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: last_changed 2024-03-30T13:52:42.231032+0000 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: created 2024-03-30T13:51:09.933456+0000 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: min_mon_release 19 (squid) 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: election_strategy: 1 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: fsmap 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: mgrmap e13: y(active, since 40s) 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: overall HEALTH_OK 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:47.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:47.694 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:52:47.694 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:48 smithi102 ceph-mon[29392]: Deploying daemon mon.c on smithi102 2024-03-30T13:52:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:48.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.592+0000 7f0698ede700 1 -- 172.21.15.138:0/1112788053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 msgr2=0x7f067c005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:48.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.592+0000 7f0698ede700 1 --2- 172.21.15.138:0/1112788053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 0x7f067c005610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0684005fa0 tx=0x7f068400ffc0 comp rx=0 tx=0).stop 2024-03-30T13:52:48.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.592+0000 7f0698ede700 1 -- 172.21.15.138:0/1112788053 shutdown_connections 2024-03-30T13:52:48.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.592+0000 7f0698ede700 1 --2- 172.21.15.138:0/1112788053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 0x7f067c005610 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:48.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.592+0000 7f0698ede700 1 -- 172.21.15.138:0/1112788053 >> 172.21.15.138:0/1112788053 conn(0x7f06940fc6e0 msgr2=0x7f06940feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:48.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.593+0000 7f0698ede700 1 -- 172.21.15.138:0/1112788053 shutdown_connections 2024-03-30T13:52:48.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.593+0000 7f0698ede700 1 -- 172.21.15.138:0/1112788053 wait complete. 2024-03-30T13:52:48.593 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0698ede700 1 Processor -- start 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0698ede700 1 -- start start 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0698ede700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 0x7f0694195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0698ede700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 0x7f069419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0698ede700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f06941083a0 con 0x7f0694100e70 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0698ede700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0694108220 con 0x7f06941963f0 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0691d9b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 0x7f069419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:48.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0691d9b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 0x7f069419a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:44160/0 (socket says 172.21.15.138:44160) 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0691d9b700 1 -- 172.21.15.138:0/21883227 learned_addr learned my addr 172.21.15.138:0/21883227 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f069259c700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 0x7f0694195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0691d9b700 1 -- 172.21.15.138:0/21883227 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 msgr2=0x7f069419a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0691d9b700 1 -- 172.21.15.138:0/21883227 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 msgr2=0x7f069419a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0691d9b700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 0x7f069419a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.594+0000 7f0691d9b700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 0x7f069419a780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f069259c700 1 -- 172.21.15.138:0/21883227 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 msgr2=0x7f069419a780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f069259c700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 0x7f069419a780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f069259c700 1 -- 172.21.15.138:0/21883227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0694196a30 con 0x7f0694100e70 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f069259c700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 0x7f0694195eb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0684000c00 tx=0x7f0684013d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f06837fe700 1 -- 172.21.15.138:0/21883227 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0684004ac0 con 0x7f0694100e70 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f069419acc0 con 0x7f0694100e70 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f06837fe700 1 -- 172.21.15.138:0/21883227 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f0684004c60 con 0x7f0694100e70 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f06837fe700 1 -- 172.21.15.138:0/21883227 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0684011670 con 0x7f0694100e70 2024-03-30T13:52:48.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.595+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069419b220 con 0x7f0694100e70 2024-03-30T13:52:48.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.596+0000 7f06837fe700 1 -- 172.21.15.138:0/21883227 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f0684011810 con 0x7f0694100e70 2024-03-30T13:52:48.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.596+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0694102350 con 0x7f0694100e70 2024-03-30T13:52:48.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.597+0000 7f06837fe700 1 --2- 172.21.15.138:0/21883227 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0674038c10 0x7f067403b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:48.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.597+0000 7f06837fe700 1 -- 172.21.15.138:0/21883227 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (secure 0 0 0) 0x7f068404e320 con 0x7f0694100e70 2024-03-30T13:52:48.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.597+0000 7f0691d9b700 1 --2- 172.21.15.138:0/21883227 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0674038c10 0x7f067403b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:48.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.597+0000 7f0691d9b700 1 --2- 172.21.15.138:0/21883227 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0674038c10 0x7f067403b0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0688009180 tx=0x7f0688008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:48.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.602+0000 7f06837fe700 1 -- 172.21.15.138:0/21883227 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0684022070 con 0x7f0694100e70 2024-03-30T13:52:48.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.877+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0694197210 con 0x7f0694100e70 2024-03-30T13:52:48.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.878+0000 7f06837fe700 1 -- 172.21.15.138:0/21883227 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f0684019bc0 con 0x7f0694100e70 2024-03-30T13:52:48.877 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:52:48.877 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":2,"fsid":"69dd7ec8-ee9c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T13:52:42.231032Z","created":"2024-03-30T13:51:09.933456Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T13:52:48.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0674038c10 msgr2=0x7f067403b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:48.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 --2- 172.21.15.138:0/21883227 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0674038c10 0x7f067403b0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0688009180 tx=0x7f0688008040 comp rx=0 tx=0).stop 2024-03-30T13:52:48.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 msgr2=0x7f0694195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 0x7f0694195eb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0684000c00 tx=0x7f0684013d80 comp rx=0 tx=0).stop 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 shutdown_connections 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 --2- 172.21.15.138:0/21883227 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0674038c10 0x7f067403b0d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0694100e70 0x7f0694195eb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 --2- 172.21.15.138:0/21883227 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f06941963f0 0x7f069419a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 >> 172.21.15.138:0/21883227 conn(0x7f06940fc6e0 msgr2=0x7f06940fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 shutdown_connections 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:48.880+0000 7f0698ede700 1 -- 172.21.15.138:0/21883227 wait complete. 2024-03-30T13:52:48.880 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 2 2024-03-30T13:52:49.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/21883227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:52:49.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:49 smithi138 ceph-mon[33397]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:49.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/21883227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:52:49.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:49 smithi102 ceph-mon[29392]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:50.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 systemd[1]: Starting Ceph mon.c for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:52:50.496 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T13:52:50.496 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 podman[37697]: 2024-03-30 13:52:50.37038391 +0000 UTC m=+0.018333934 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 podman[37697]: 2024-03-30 13:52:50.478049784 +0000 UTC m=+0.125999797 container create f2c2a15c7568bb631b79415f6827aeeb541c6b956995174cf487c471eca0fd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 podman[37697]: 2024-03-30 13:52:50.680410644 +0000 UTC m=+0.328360658 container init f2c2a15c7568bb631b79415f6827aeeb541c6b956995174cf487c471eca0fd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 podman[37697]: 2024-03-30 13:52:50.683439815 +0000 UTC m=+0.331389829 container start f2c2a15c7568bb631b79415f6827aeeb541c6b956995174cf487c471eca0fd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-4e50bbc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 2 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pidfile_write: ignore empty --pid-file 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: load: jerasure load: lrc 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: RocksDB version: 7.9.2 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Git sha 0 2024-03-30T13:52:50.717 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: DB SUMMARY 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: DB Session ID: 9E13BCS0BE052L8LNB25 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: CURRENT file: CURRENT 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.error_if_exists: 0 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.create_if_missing: 0 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.paranoid_checks: 1 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T13:52:50.718 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.env: 0x5571ce7eac60 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.info_log: 0x5571d04b3ae0 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.statistics: (nil) 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.use_fsync: 0 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_log_file_size: 0 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.allow_fallocate: 1 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T13:52:50.719 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.use_direct_reads: 0 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.db_log_dir: 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.wal_dir: 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.write_buffer_manager: 0x5571d04861e0 2024-03-30T13:52:50.720 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T13:52:50.721 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T13:52:50.721 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T13:52:50.721 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.unordered_write: 0 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T13:52:50.722 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.row_cache: None 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.wal_filter: None 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.two_write_queues: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.wal_compression: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.atomic_flush: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.log_readahead_size: 0 2024-03-30T13:52:50.723 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_background_jobs: 2 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_background_compactions: -1 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_subcompactions: 1 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T13:52:50.724 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_open_files: -1 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_background_flushes: -1 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Compression algorithms supported: 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kZSTD supported: 0 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kXpressCompression supported: 0 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kZlibCompression supported: 1 2024-03-30T13:52:50.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kSnappyCompression supported: 1 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kLZ4Compression supported: 1 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: kBZip2Compression supported: 0 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.merge_operator: 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_filter: None 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T13:52:50.726 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5571d04b3be0) 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_type: 0 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_index_type: 0 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_shortening: 1 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: checksum: 4 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: no_block_cache: 0 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache: 0x5571d04b7090 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_options: 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: capacity : 536870912 2024-03-30T13:52:50.727 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_shard_bits : 4 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: strict_capacity_limit : 0 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_compressed: (nil) 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: persistent_cache: (nil) 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size: 4096 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size_deviation: 10 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_restart_interval: 16 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_block_restart_interval: 1 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: metadata_block_size: 4096 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: partition_filters: 0 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: use_delta_encoding: 1 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: filter_policy: bloomfilter 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: whole_key_filtering: 1 2024-03-30T13:52:50.728 INFO:journalctl@ceph.mon.c.smithi102.stdout: verify_compression: 0 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: format_version: 5 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_index_compression: 1 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_align: 0 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_auto_readahead_size: 262144 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: prepopulate_block_cache: 0 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: initial_auto_readahead_size: 8192 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression: NoCompression 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T13:52:50.729 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.num_levels: 7 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:52:50.730 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T13:52:50.731 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T13:52:50.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T13:52:50.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.table_properties_collectors: 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.inplace_update_support: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.bloom_locality: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.max_successive_merges: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.ttl: 2592000 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T13:52:50.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.enable_blob_files: false 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.min_blob_size: 0 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b7532fbb-58df-47bf-942c-1ef71b47d1ce 2024-03-30T13:52:50.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711806770714476, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711806770714856, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711806770, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b7532fbb-58df-47bf-942c-1ef71b47d1ce", "db_session_id": "9E13BCS0BE052L8LNB25", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711806770714936, "job": 1, "event": "recovery_finished"} 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5571d05a8000 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: DB pointer 0x5571d0592000 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** DB Stats ** 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:52:50.736 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T13:52:50.737 INFO:journalctl@ceph.mon.c.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative compaction: 0.00 GB write, 0.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval compaction: 0.00 GB write, 0.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache BinnedLRUCache@0x5571d04b7090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-30T13:52:50.738 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-30T13:52:50.739 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T13:52:50.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-30T13:52:50.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: using public_addrv [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-03-30T13:52:50.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: starting mon.c rank -1 at public addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] at bind addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:50.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(???) e0 preinit fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:50.742 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).mds e1 new map 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: e1 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: legacy client fscid: -1 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout: No filesystems configured 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T13:52:51.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:52:51.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3001285721' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3001285721' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi138:172.21.15.138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi138:172.21.15.138=b;count:3 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: Deploying daemon mon.b on smithi138 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.a calling monitor election 2024-03-30T13:52:51.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.b calling monitor election 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T13:52:51.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: monmap epoch 2 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: last_changed 2024-03-30T13:52:42.231032+0000 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: created 2024-03-30T13:51:09.933456+0000 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: min_mon_release 19 (squid) 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: election_strategy: 1 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: fsmap 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mgrmap e13: y(active, since 40s) 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: overall HEALTH_OK 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:51.197 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: Deploying daemon mon.c on smithi102 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/21883227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 ceph-mon[37717]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 bash[37697]: f2c2a15c7568bb631b79415f6827aeeb541c6b956995174cf487c471eca0fd78 2024-03-30T13:52:51.198 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:50 smithi102 systemd[1]: Started Ceph mon.c for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:52:56.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:52:56.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: mon.a calling monitor election 2024-03-30T13:52:56.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:56.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: mon.b calling monitor election 2024-03-30T13:52:56.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: mon.c calling monitor election 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: monmap epoch 3 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: last_changed 2024-03-30T13:52:50.734785+0000 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: created 2024-03-30T13:51:09.933456+0000 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: min_mon_release 19 (squid) 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: election_strategy: 1 2024-03-30T13:52:56.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: fsmap 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: mgrmap e13: y(active, since 48s) 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: overall HEALTH_OK 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.190 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:52:56.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: mon.a calling monitor election 2024-03-30T13:52:56.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:56.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: mon.b calling monitor election 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: mon.c calling monitor election 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: monmap epoch 3 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:56.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: last_changed 2024-03-30T13:52:50.734785+0000 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: created 2024-03-30T13:51:09.933456+0000 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: min_mon_release 19 (squid) 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: election_strategy: 1 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: fsmap 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: mgrmap e13: y(active, since 48s) 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: overall HEALTH_OK 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: mon.a calling monitor election 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: mon.b calling monitor election 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: mon.c calling monitor election 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: monmap epoch 3 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: last_changed 2024-03-30T13:52:50.734785+0000 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: created 2024-03-30T13:51:09.933456+0000 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: min_mon_release 19 (squid) 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: election_strategy: 1 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: 1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: 2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: fsmap 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: mgrmap e13: y(active, since 48s) 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: overall HEALTH_OK 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.516+0000 7f22d8264700 1 -- 172.21.15.138:0/1863269512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc0035c0 msgr2=0x7f22bc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.516+0000 7f22d8264700 1 --2- 172.21.15.138:0/1863269512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc0035c0 0x7f22bc005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f22d0100b30 tx=0x7f22cc00fe60 comp rx=0 tx=0).stop 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.516+0000 7f22d8264700 1 -- 172.21.15.138:0/1863269512 shutdown_connections 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.516+0000 7f22d8264700 1 --2- 172.21.15.138:0/1863269512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc0035c0 0x7f22bc005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.516+0000 7f22d8264700 1 -- 172.21.15.138:0/1863269512 >> 172.21.15.138:0/1863269512 conn(0x7f22d00fc6f0 msgr2=0x7f22d00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.516+0000 7f22d8264700 1 -- 172.21.15.138:0/1863269512 shutdown_connections 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.517+0000 7f22d8264700 1 -- 172.21.15.138:0/1863269512 wait complete. 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.517+0000 7f22d8264700 1 Processor -- start 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.517+0000 7f22d8264700 1 -- start start 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d8264700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 0x7f22d0102900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d8264700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22d0102e40 0x7f22d01032c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:56.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d8264700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 0x7f22d01a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:56.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d8264700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f22d0108350 con 0x7f22d0102e40 2024-03-30T13:52:56.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d8264700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01081d0 con 0x7f22d01a3300 2024-03-30T13:52:56.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d8264700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f22d01084d0 con 0x7f22bc0035c0 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22d0102e40 0x7f22d01032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6000700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 0x7f22d0102900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22d0102e40 0x7f22d01032c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.138:39792/0 (socket says 172.21.15.138:39792) 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6000700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 0x7f22d0102900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.138:39760/0 (socket says 172.21.15.138:39760) 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 -- 172.21.15.138:0/275034987 learned_addr learned my addr 172.21.15.138:0/275034987 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6801700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 0x7f22d01a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6000700 1 -- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 msgr2=0x7f22d0102900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-30T13:52:56.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6000700 1 -- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 msgr2=0x7f22d0102900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6000700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 0x7f22d0102900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6000700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 0x7f22d0102900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6801700 1 -- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 msgr2=0x7f22d01a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6801700 1 -- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 msgr2=0x7f22d01a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6801700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 0x7f22d01a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d6801700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 0x7f22d01a56f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 -- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 msgr2=0x7f22d0102900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 0x7f22d0102900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 -- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 msgr2=0x7f22d01a56f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 0x7f22d01a56f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 -- 172.21.15.138:0/275034987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d01a5e20 con 0x7f22d0102e40 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.518+0000 7f22d57ff700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22d0102e40 0x7f22d01032c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22c000b780 tx=0x7f22c000bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.519+0000 7f22c6ffd700 1 -- 172.21.15.138:0/275034987 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0004230 con 0x7f22d0102e40 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.519+0000 7f22c6ffd700 1 -- 172.21.15.138:0/275034987 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f22c00043d0 con 0x7f22d0102e40 2024-03-30T13:52:56.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.519+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d01a6110 con 0x7f22d0102e40 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.519+0000 7f22c6ffd700 1 -- 172.21.15.138:0/275034987 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0011b50 con 0x7f22d0102e40 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.519+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d01a6650 con 0x7f22d0102e40 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.520+0000 7f22c6ffd700 1 -- 172.21.15.138:0/275034987 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f22c001a400 con 0x7f22d0102e40 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.520+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d0063870 con 0x7f22d0102e40 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.520+0000 7f22c6ffd700 1 --2- 172.21.15.138:0/275034987 >> v2:172.21.15.102:6800/3154847229 conn(0x7f22b40392f0 0x7f22b403b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.520+0000 7f22c6ffd700 1 -- 172.21.15.138:0/275034987 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (secure 0 0 0) 0x7f22c004ba30 con 0x7f22d0102e40 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.520+0000 7f22d6000700 1 --2- 172.21.15.138:0/275034987 >> v2:172.21.15.102:6800/3154847229 conn(0x7f22b40392f0 0x7f22b403b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:52:56.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.520+0000 7f22d6000700 1 --2- 172.21.15.138:0/275034987 >> v2:172.21.15.102:6800/3154847229 conn(0x7f22b40392f0 0x7f22b403b7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00fa10 tx=0x7f22cc005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:52:56.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.525+0000 7f22c6ffd700 1 -- 172.21.15.138:0/275034987 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f22c001a7f0 con 0x7f22d0102e40 2024-03-30T13:52:56.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.810+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f22d002db30 con 0x7f22d0102e40 2024-03-30T13:52:56.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.810+0000 7f22c6ffd700 1 -- 172.21.15.138:0/275034987 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7f22c0018750 con 0x7f22d0102e40 2024-03-30T13:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-30T13:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":3,"fsid":"69dd7ec8-ee9c-11ee-b647-cb9ed24678a4","modified":"2024-03-30T13:52:50.734785Z","created":"2024-03-30T13:51:09.933456Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3301","nonce":0},{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.812+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 >> v2:172.21.15.102:6800/3154847229 conn(0x7f22b40392f0 msgr2=0x7f22b403b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.812+0000 7f22d8264700 1 --2- 172.21.15.138:0/275034987 >> v2:172.21.15.102:6800/3154847229 conn(0x7f22b40392f0 0x7f22b403b7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00fa10 tx=0x7f22cc005a90 comp rx=0 tx=0).stop 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.812+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22d0102e40 msgr2=0x7f22d01032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.812+0000 7f22d8264700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22d0102e40 0x7f22d01032c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22c000b780 tx=0x7f22c000bb40 comp rx=0 tx=0).stop 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 shutdown_connections 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 --2- 172.21.15.138:0/275034987 >> v2:172.21.15.102:6800/3154847229 conn(0x7f22b40392f0 0x7f22b403b7b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22d01a3300 0x7f22d01a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:56.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f22bc0035c0 0x7f22d0102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:56.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 --2- 172.21.15.138:0/275034987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22d0102e40 0x7f22d01032c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:52:56.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 >> 172.21.15.138:0/275034987 conn(0x7f22d00fc6f0 msgr2=0x7f22d018e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:52:56.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 shutdown_connections 2024-03-30T13:52:56.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:52:56.813+0000 7f22d8264700 1 -- 172.21.15.138:0/275034987 wait complete. 2024-03-30T13:52:56.813 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 3 2024-03-30T13:52:56.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:56.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:56.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:56.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:56.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:56 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:57.152 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:56 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:57.152 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:56 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:57.152 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:56 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:52:57.152 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:56 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:52:57.821 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-30T13:52:57.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-30T13:52:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/275034987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/275034987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:52:58.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:52:58.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:58.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/275034987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T13:52:58.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:52:58.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:52:58.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:52:58.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:52:58.189 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:52:59.115 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[37717]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-30T13:52:59.115 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[37717]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:52:59.115 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[37717]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:59.115 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[37717]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:59.115 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[37717]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T13:52:59.115 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[37717]: Reconfiguring daemon mon.a on smithi102 2024-03-30T13:52:59.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[29392]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-30T13:52:59.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[29392]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:52:59.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[29392]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:59.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[29392]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:59.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[29392]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T13:52:59.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:58 smithi102 ceph-mon[29392]: Reconfiguring daemon mon.a on smithi102 2024-03-30T13:52:59.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:58 smithi138 ceph-mon[33397]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-30T13:52:59.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:58 smithi138 ceph-mon[33397]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:52:59.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:58 smithi138 ceph-mon[33397]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:59.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:58 smithi138 ceph-mon[33397]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:52:59.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:58 smithi138 ceph-mon[33397]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T13:52:59.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:58 smithi138 ceph-mon[33397]: Reconfiguring daemon mon.a on smithi102 2024-03-30T13:53:00.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:52:59 smithi102 ceph-mon[29392]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:00.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:52:59 smithi102 ceph-mon[37717]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:00.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:52:59 smithi138 ceph-mon[33397]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:00.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.222+0000 7f93dde12700 1 -- 172.21.15.102:0/2377297884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 msgr2=0x7f93d81031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:00.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 --2- 172.21.15.102:0/2377297884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d81031a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009910 tx=0x7f93c8009c20 comp rx=0 tx=0).stop 2024-03-30T13:53:00.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 -- 172.21.15.102:0/2377297884 shutdown_connections 2024-03-30T13:53:00.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 --2- 172.21.15.102:0/2377297884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d81031a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:00.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 -- 172.21.15.102:0/2377297884 >> 172.21.15.102:0/2377297884 conn(0x7f93d80fc5f0 msgr2=0x7f93d80fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:00.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 -- 172.21.15.102:0/2377297884 shutdown_connections 2024-03-30T13:53:00.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 -- 172.21.15.102:0/2377297884 wait complete. 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 Processor -- start 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.223+0000 7f93dde12700 1 -- start start 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93dde12700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d8100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93dde12700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d8100f60 0x7f93d81013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93dde12700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f93d81a3290 0x7f93d81a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93dde12700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f93d810b940 con 0x7f93d8102dc0 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93dde12700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f93d810b7c0 con 0x7f93d81a3290 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93dde12700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f93d810bac0 con 0x7f93d8100f60 2024-03-30T13:53:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d8100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d8100a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60484/0 (socket says 172.21.15.102:60484) 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 -- 172.21.15.102:0/701313965 learned_addr learned my addr 172.21.15.102:0/701313965 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d6ffd700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d8100f60 0x7f93d81013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 -- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d8100f60 msgr2=0x7f93d81013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d8100f60 0x7f93d81013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 -- 172.21.15.102:0/701313965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f93d81a3290 msgr2=0x7f93d81a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d7fff700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f93d81a3290 0x7f93d81a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f93d81a3290 0x7f93d81a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 -- 172.21.15.102:0/701313965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d81a5db0 con 0x7f93d8102dc0 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d7fff700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f93d81a3290 0x7f93d81a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d77fe700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d8100a20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f93c8004bf0 tx=0x7f93c8004c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d4ff9700 1 -- 172.21.15.102:0/701313965 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8027070 con 0x7f93d8102dc0 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d4ff9700 1 -- 172.21.15.102:0/701313965 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f93c8005da0 con 0x7f93d8102dc0 2024-03-30T13:53:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93d4ff9700 1 -- 172.21.15.102:0/701313965 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8022d10 con 0x7f93d8102dc0 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.224+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d81a6040 con 0x7f93d8102dc0 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.225+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d81a64d0 con 0x7f93d8102dc0 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.225+0000 7f93d4ff9700 1 -- 172.21.15.102:0/701313965 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f93c802a070 con 0x7f93d8102dc0 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.225+0000 7f93d4ff9700 1 --2- 172.21.15.102:0/701313965 >> v2:172.21.15.102:6800/3154847229 conn(0x7f93c00392f0 0x7f93c003b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.225+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d8063870 con 0x7f93d8102dc0 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.225+0000 7f93d4ff9700 1 -- 172.21.15.102:0/701313965 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (secure 0 0 0) 0x7f93c804cf10 con 0x7f93d8102dc0 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.225+0000 7f93d6ffd700 1 --2- 172.21.15.102:0/701313965 >> v2:172.21.15.102:6800/3154847229 conn(0x7f93c00392f0 0x7f93c003b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.225+0000 7f93d6ffd700 1 --2- 172.21.15.102:0/701313965 >> v2:172.21.15.102:6800/3154847229 conn(0x7f93c00392f0 0x7f93c003b7b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009ad0 tx=0x7f93cc009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.231+0000 7f93d4ff9700 1 -- 172.21.15.102:0/701313965 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f93c8013070 con 0x7f93d8102dc0 2024-03-30T13:53:00.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.430+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f93d802db30 con 0x7f93d8102dc0 2024-03-30T13:53:00.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.430+0000 7f93d4ff9700 1 -- 172.21.15.102:0/701313965 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f93c802adf0 con 0x7f93d8102dc0 2024-03-30T13:53:00.429 INFO:teuthology.orchestra.run.smithi102.stdout:# minimal ceph.conf for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:53:00.429 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-03-30T13:53:00.429 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:53:00.429 INFO:teuthology.orchestra.run.smithi102.stdout: mon_host = [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 >> v2:172.21.15.102:6800/3154847229 conn(0x7f93c00392f0 msgr2=0x7f93c003b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 --2- 172.21.15.102:0/701313965 >> v2:172.21.15.102:6800/3154847229 conn(0x7f93c00392f0 0x7f93c003b7b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009ad0 tx=0x7f93cc009210 comp rx=0 tx=0).stop 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 msgr2=0x7f93d8100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d8100a20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f93c8004bf0 tx=0x7f93c8004c20 comp rx=0 tx=0).stop 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 shutdown_connections 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 --2- 172.21.15.102:0/701313965 >> v2:172.21.15.102:6800/3154847229 conn(0x7f93c00392f0 0x7f93c003b7b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d8102dc0 0x7f93d8100a20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:00.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f93d81a3290 0x7f93d81a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:00.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 --2- 172.21.15.102:0/701313965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d8100f60 0x7f93d81013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:00.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.432+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 >> 172.21.15.102:0/701313965 conn(0x7f93d80fc5f0 msgr2=0x7f93d80fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:00.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.433+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 shutdown_connections 2024-03-30T13:53:00.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:00.433+0000 7f93dde12700 1 -- 172.21.15.102:0/701313965 wait complete. 2024-03-30T13:53:01.069 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/701313965' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/701313965' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:53:01.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:53:01.071 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:01.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/701313965' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:01.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:01.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:01.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:53:01.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:53:01.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:01.372 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-30T13:53:01.372 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:53:01.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T13:53:01.402 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:53:01.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:53:01.469 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:53:01.470 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T13:53:01.504 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:53:01.504 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:53:01.574 INFO:tasks.cephadm:Adding mgr.y on smithi102 2024-03-30T13:53:01.575 INFO:tasks.cephadm:Adding mgr.x on smithi138 2024-03-30T13:53:01.575 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '2;smithi102=y;smithi138=x' 2024-03-30T13:53:01.850 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:53:01.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:01 smithi102 ceph-mon[29392]: Reconfiguring mon.c (monmap changed)... 2024-03-30T13:53:01.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:01 smithi102 ceph-mon[29392]: Reconfiguring daemon mon.c on smithi102 2024-03-30T13:53:01.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:01 smithi102 ceph-mon[29392]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:01 smithi102 ceph-mon[37717]: Reconfiguring mon.c (monmap changed)... 2024-03-30T13:53:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:01 smithi102 ceph-mon[37717]: Reconfiguring daemon mon.c on smithi102 2024-03-30T13:53:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:01 smithi102 ceph-mon[37717]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:02.138 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:01 smithi138 ceph-mon[33397]: Reconfiguring mon.c (monmap changed)... 2024-03-30T13:53:02.138 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:01 smithi138 ceph-mon[33397]: Reconfiguring daemon mon.c on smithi102 2024-03-30T13:53:02.138 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:01 smithi138 ceph-mon[33397]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:03.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.374+0000 7fb513a40700 1 -- 172.21.15.138:0/4272698142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c102280 msgr2=0x7fb50c1026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:03.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.374+0000 7fb513a40700 1 --2- 172.21.15.138:0/4272698142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c102280 0x7fb50c1026a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb508009910 tx=0x7fb508009c20 comp rx=0 tx=0).stop 2024-03-30T13:53:03.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.374+0000 7fb513a40700 1 -- 172.21.15.138:0/4272698142 shutdown_connections 2024-03-30T13:53:03.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.374+0000 7fb513a40700 1 --2- 172.21.15.138:0/4272698142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c102280 0x7fb50c1026a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:03.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.374+0000 7fb513a40700 1 -- 172.21.15.138:0/4272698142 >> 172.21.15.138:0/4272698142 conn(0x7fb50c0fb990 msgr2=0x7fb50c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:03.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.374+0000 7fb513a40700 1 -- 172.21.15.138:0/4272698142 shutdown_connections 2024-03-30T13:53:03.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.374+0000 7fb513a40700 1 -- 172.21.15.138:0/4272698142 wait complete. 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 Processor -- start 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 -- start start 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb50c102280 0x7fb50c196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb50c1971e0 0x7fb50c19c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c197660 0x7fb50c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c106370 con 0x7fb50c197660 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1061f0 con 0x7fb50c1971e0 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb513a40700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb50c1064f0 con 0x7fb50c102280 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb510fdb700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb50c1971e0 0x7fb50c19c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb510fdb700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb50c1971e0 0x7fb50c19c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:58848/0 (socket says 172.21.15.138:58848) 2024-03-30T13:53:03.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.375+0000 7fb510fdb700 1 -- 172.21.15.138:0/345242291 learned_addr learned my addr 172.21.15.138:0/345242291 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb5117dc700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb50c102280 0x7fb50c196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb511fdd700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c197660 0x7fb50c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb510fdb700 1 -- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb50c102280 msgr2=0x7fb50c196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb510fdb700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb50c102280 0x7fb50c196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb510fdb700 1 -- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c197660 msgr2=0x7fb50c19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb510fdb700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c197660 0x7fb50c19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb510fdb700 1 -- 172.21.15.138:0/345242291 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c19f0f0 con 0x7fb50c1971e0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb510fdb700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb50c1971e0 0x7fb50c19c240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00b780 tx=0x7fb4fc00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb5027fc700 1 -- 172.21.15.138:0/345242291 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc004230 con 0x7fb50c1971e0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb50c19f3e0 con 0x7fb50c1971e0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb5027fc700 1 -- 172.21.15.138:0/345242291 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fb4fc0043d0 con 0x7fb50c1971e0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb5027fc700 1 -- 172.21.15.138:0/345242291 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc011b40 con 0x7fb50c1971e0 2024-03-30T13:53:03.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.376+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb50c19f9f0 con 0x7fb50c1971e0 2024-03-30T13:53:03.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.377+0000 7fb5027fc700 1 -- 172.21.15.138:0/345242291 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7fb4fc011ce0 con 0x7fb50c1971e0 2024-03-30T13:53:03.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.377+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb50c063870 con 0x7fb50c1971e0 2024-03-30T13:53:03.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.377+0000 7fb5027fc700 1 --2- 172.21.15.138:0/345242291 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb4f80392f0 0x7fb4f803b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:03.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.377+0000 7fb5027fc700 1 -- 172.21.15.138:0/345242291 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (secure 0 0 0) 0x7fb4fc04ba30 con 0x7fb50c1971e0 2024-03-30T13:53:03.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.378+0000 7fb5117dc700 1 --2- 172.21.15.138:0/345242291 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb4f80392f0 0x7fb4f803b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:03.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.378+0000 7fb5117dc700 1 --2- 172.21.15.138:0/345242291 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb4f80392f0 0x7fb4f803b7b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb508000c00 tx=0x7fb50801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:03.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.382+0000 7fb5027fc700 1 -- 172.21.15.138:0/345242291 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb4fc0202a0 con 0x7fb50c1971e0 2024-03-30T13:53:03.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.597+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi138=x", "target": ["mon-mgr", ""]}) -- 0x7fb50c062680 con 0x7fb4f80392f0 2024-03-30T13:53:03.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.603+0000 7fb5027fc700 1 -- 172.21.15.138:0/345242291 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb50c062680 con 0x7fb4f80392f0 2024-03-30T13:53:03.603 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mgr update... 2024-03-30T13:53:03.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb4f80392f0 msgr2=0x7fb4f803b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:03.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 --2- 172.21.15.138:0/345242291 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb4f80392f0 0x7fb4f803b7b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb508000c00 tx=0x7fb50801a040 comp rx=0 tx=0).stop 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb50c1971e0 msgr2=0x7fb50c19c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb50c1971e0 0x7fb50c19c240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00b780 tx=0x7fb4fc00ba90 comp rx=0 tx=0).stop 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 shutdown_connections 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 --2- 172.21.15.138:0/345242291 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb4f80392f0 0x7fb4f803b7b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb50c197660 0x7fb50c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb50c102280 0x7fb50c196ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 --2- 172.21.15.138:0/345242291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb50c1971e0 0x7fb50c19c240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 >> 172.21.15.138:0/345242291 conn(0x7fb50c0fb990 msgr2=0x7fb50c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 shutdown_connections 2024-03-30T13:53:03.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:53:03.606+0000 7fb513a40700 1 -- 172.21.15.138:0/345242291 wait complete. 2024-03-30T13:53:03.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:03.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:03.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: Reconfiguring mon.b (monmap changed)... 2024-03-30T13:53:03.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:53:03.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:53:03.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:03.799 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: Reconfiguring daemon mon.b on smithi138 2024-03-30T13:53:03.799 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:03 smithi138 ceph-mon[33397]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:03.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: Reconfiguring mon.b (monmap changed)... 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: Reconfiguring daemon mon.b on smithi138 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[29392]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: Reconfiguring mon.b (monmap changed)... 2024-03-30T13:53:03.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T13:53:03.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T13:53:03.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:03.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: Reconfiguring daemon mon.b on smithi138 2024-03-30T13:53:03.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:03 smithi102 ceph-mon[37717]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:04.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:04 smithi138 ceph-mon[33397]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi138=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:04.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:04 smithi138 ceph-mon[33397]: Saving service mgr spec with placement smithi102=y;smithi138=x;count:2 2024-03-30T13:53:04.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:04 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:04.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:04 smithi102 ceph-mon[29392]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi138=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:04.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:04 smithi102 ceph-mon[29392]: Saving service mgr spec with placement smithi102=y;smithi138=x;count:2 2024-03-30T13:53:04.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:04 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:04.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:04 smithi102 ceph-mon[37717]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi138=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:04.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:04 smithi102 ceph-mon[37717]: Saving service mgr spec with placement smithi102=y;smithi138=x;count:2 2024-03-30T13:53:04.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:04 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.058 DEBUG:teuthology.orchestra.run.smithi138:mgr.x> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.x.service 2024-03-30T13:53:05.060 INFO:tasks.cephadm:Deploying OSDs... 2024-03-30T13:53:05.060 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:53:05.060 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T13:53:05.078 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T13:53:05.078 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-30T13:53:05.134 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T13:53:05.134 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:05.135 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-30T13:53:05.135 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:05.135 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:05.135 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-30 13:52:50.187537499 +0000 2024-03-30T13:53:05.135 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-30 13:45:03.068042485 +0000 2024-03-30T13:53:05.135 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-30 13:45:03.068042485 +0000 2024-03-30T13:53:05.135 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-30 13:45:03.068042485 +0000 2024-03-30T13:53:05.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T13:53:05.203 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-30T13:53:05.204 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-30T13:53:05.204 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000225836 s, 2.3 MB/s 2024-03-30T13:53:05.205 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T13:53:05.262 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-30T13:53:05.317 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T13:53:05.317 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:05.318 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-30T13:53:05.318 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:05.318 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:05.318 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-30 13:52:50.188537478 +0000 2024-03-30T13:53:05.318 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-30 13:45:03.414035483 +0000 2024-03-30T13:53:05.318 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-30 13:45:03.414035483 +0000 2024-03-30T13:53:05.318 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-30 13:45:03.414035483 +0000 2024-03-30T13:53:05.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T13:53:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-30T13:53:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-30T13:53:05.386 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000200323 s, 2.6 MB/s 2024-03-30T13:53:05.387 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T13:53:05.446 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-30T13:53:05.501 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-30 13:52:50.188537478 +0000 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-30 13:45:03.767028340 +0000 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-30 13:45:03.767028340 +0000 2024-03-30T13:53:05.502 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-30 13:45:03.767028340 +0000 2024-03-30T13:53:05.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T13:53:05.568 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-30T13:53:05.569 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-30T13:53:05.569 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000195215 s, 2.6 MB/s 2024-03-30T13:53:05.570 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T13:53:05.627 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-30T13:53:05.682 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-30 13:52:50.188537478 +0000 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-30 13:45:04.115021269 +0000 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-30 13:45:04.115021269 +0000 2024-03-30T13:53:05.683 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-30 13:45:04.115021269 +0000 2024-03-30T13:53:05.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T13:53:05.750 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-30T13:53:05.750 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-30T13:53:05.750 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000193685 s, 2.6 MB/s 2024-03-30T13:53:05.751 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T13:53:05.807 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:53:05.807 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T13:53:05.823 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T13:53:05.823 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-30 13:52:41.157743520 +0000 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-30 13:45:41.981281576 +0000 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-30 13:45:41.981281576 +0000 2024-03-30T13:53:05.878 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-30 13:45:41.981281576 +0000 2024-03-30T13:53:05.879 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T13:53:05.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:05.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: Deploying daemon mgr.x on smithi138 2024-03-30T13:53:05.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:05 smithi138 ceph-mon[33397]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:05.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: Deploying daemon mgr.x on smithi138 2024-03-30T13:53:05.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[29392]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: Deploying daemon mgr.x on smithi138 2024-03-30T13:53:05.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:05 smithi102 ceph-mon[37717]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:05.944 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-30T13:53:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-30T13:53:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000259694 s, 2.0 MB/s 2024-03-30T13:53:05.946 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T13:53:06.001 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-03-30T13:53:06.056 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T13:53:06.056 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:06.057 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-30T13:53:06.057 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:06.057 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:06.057 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-30 13:52:41.158743497 +0000 2024-03-30T13:53:06.057 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-30 13:45:42.339273517 +0000 2024-03-30T13:53:06.057 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-30 13:45:42.339273517 +0000 2024-03-30T13:53:06.057 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-30 13:45:42.339273517 +0000 2024-03-30T13:53:06.058 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T13:53:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-30T13:53:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-30T13:53:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000209244 s, 2.4 MB/s 2024-03-30T13:53:06.123 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T13:53:06.179 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-03-30T13:53:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T13:53:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:06.233 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-30T13:53:06.233 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:06.233 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:06.233 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-30 13:52:41.158743497 +0000 2024-03-30T13:53:06.234 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-30 13:45:42.671266044 +0000 2024-03-30T13:53:06.234 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-30 13:45:42.671266044 +0000 2024-03-30T13:53:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-30 13:45:42.671266044 +0000 2024-03-30T13:53:06.234 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T13:53:06.299 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-30T13:53:06.299 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-30T13:53:06.299 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.0001769 s, 2.9 MB/s 2024-03-30T13:53:06.300 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T13:53:06.357 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-03-30 13:52:41.158743497 +0000 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-03-30 13:45:43.023258121 +0000 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-03-30 13:45:43.023258121 +0000 2024-03-30T13:53:06.420 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-03-30 13:45:43.023258121 +0000 2024-03-30T13:53:06.421 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T13:53:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-03-30T13:53:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-03-30T13:53:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000233487 s, 2.2 MB/s 2024-03-30T13:53:06.528 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T13:53:06.543 INFO:tasks.cephadm:Deploying osd.0 on smithi102 with /dev/vg_nvme/lv_4... 2024-03-30T13:53:06.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T13:53:06.775 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:06.931 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:06 smithi138 systemd[1]: Starting Ceph mgr.x for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:53:07.188 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:06 smithi138 podman[35251]: 2024-03-30 13:53:06.931907195 +0000 UTC m=+0.107343924 container create c79e20d192f94021fe6cdf733f28d586b81d4eb7ee4bc6406cbb0beae3ed5240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x, RELEASE=squid-4e50bbc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-30T13:53:07.188 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:06 smithi138 podman[35251]: 2024-03-30 13:53:06.842162175 +0000 UTC m=+0.017598865 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:53:07.188 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:07 smithi138 podman[35251]: 2024-03-30 13:53:07.150701353 +0000 UTC m=+0.326138050 container init c79e20d192f94021fe6cdf733f28d586b81d4eb7ee4bc6406cbb0beae3ed5240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-30T13:53:07.188 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:07 smithi138 podman[35251]: 2024-03-30 13:53:07.153481167 +0000 UTC m=+0.328917862 container start c79e20d192f94021fe6cdf733f28d586b81d4eb7ee4bc6406cbb0beae3ed5240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-30T13:53:07.188 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:07 smithi138 ceph-mgr[35270]: -- 172.21.15.138:0/1162866472 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558f2fc465a0 con 0x558f2fc1ac00 2024-03-30T13:53:07.687 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:07 smithi138 bash[35251]: c79e20d192f94021fe6cdf733f28d586b81d4eb7ee4bc6406cbb0beae3ed5240 2024-03-30T13:53:07.687 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:07 smithi138 systemd[1]: Started Ceph mgr.x for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:53:08.055 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:07.717+0000 7fbfa3ab8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:53:08.055 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:07.814+0000 7fbfa3ab8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:53:08.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[37717]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[29392]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.102 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:08.393 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-mon[33397]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:08.393 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.394 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.394 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.394 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:08.394 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:08.645 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T13:53:08.646 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T13:53:08.646 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-30T13:53:08.646 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-30T13:53:08.646 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115488 s, 908 MB/s 2024-03-30T13:53:08.646 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-30T13:53:08.687 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:08.504+0000 7fbfa3ab8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:53:08.687 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:08.602+0000 7fbfa3ab8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:53:09.187 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:08.755+0000 7fbfa3ab8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:53:09.187 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:08.869+0000 7fbfa3ab8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:53:09.187 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:08.959+0000 7fbfa3ab8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:53:09.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2024-03-30T13:53:09.529 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:09.232+0000 7fbfa3ab8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:53:09.529 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:09.345+0000 7fbfa3ab8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:53:09.529 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:09.433+0000 7fbfa3ab8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:53:09.529 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:09.529+0000 7fbfa3ab8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:53:09.817 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:09.710+0000 7fbfa3ab8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:53:09.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:09.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:09.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:09.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:09.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:09.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: Reconfiguring daemon mgr.y on smithi102 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[37717]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T13:53:09.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:53:09.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:53:09.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:09.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: Reconfiguring daemon mgr.y on smithi102 2024-03-30T13:53:09.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:09 smithi102 ceph-mon[29392]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:09.997 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:10.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:10.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: Reconfiguring daemon mgr.y on smithi102 2024-03-30T13:53:10.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:09 smithi138 ceph-mon[33397]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:10.937 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:10 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:10.685+0000 7fbfa3ab8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:53:10.937 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:10 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:10.814+0000 7fbfa3ab8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:53:11.437 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:11.161+0000 7fbfa3ab8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:53:11.437 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:11.251+0000 7fbfa3ab8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:53:11.918 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:11.446+0000 7fbfa3ab8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:53:11.918 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:11.630+0000 7fbfa3ab8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:53:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 -- 172.21.15.102:0/2908392490 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c102d00 msgr2=0x7f757c1050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2908392490 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c102d00 0x7f757c1050f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7578009990 tx=0x7f7578009ca0 comp rx=0 tx=0).stop 2024-03-30T13:53:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 -- 172.21.15.102:0/2908392490 shutdown_connections 2024-03-30T13:53:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2908392490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f757c107f60 0x7f757c10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2908392490 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c102d00 0x7f757c1050f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2908392490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c105630 0x7f757c107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 -- 172.21.15.102:0/2908392490 >> 172.21.15.102:0/2908392490 conn(0x7f757c0fc710 msgr2=0x7f757c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:11.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 -- 172.21.15.102:0/2908392490 shutdown_connections 2024-03-30T13:53:11.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.979+0000 7f7583eb1700 1 -- 172.21.15.102:0/2908392490 wait complete. 2024-03-30T13:53:11.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 Processor -- start 2024-03-30T13:53:11.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 -- start start 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c102d00 0x7f757c197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f757c105630 0x7f757c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c107f60 0x7f757c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f757c10d590 con 0x7f757c102d00 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f757c10d410 con 0x7f757c105630 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7583eb1700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f757c10d710 con 0x7f757c107f60 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7581c4d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c102d00 0x7f757c197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f758244e700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c107f60 0x7f757c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:11.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7581c4d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c102d00 0x7f757c197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45130/0 (socket says 172.21.15.102:45130) 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7581c4d700 1 -- 172.21.15.102:0/2072008860 learned_addr learned my addr 172.21.15.102:0/2072008860 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f758144c700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f757c105630 0x7f757c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f758244e700 1 -- 172.21.15.102:0/2072008860 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f757c105630 msgr2=0x7f757c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f758244e700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f757c105630 0x7f757c1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f758244e700 1 -- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c102d00 msgr2=0x7f757c197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f758244e700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c102d00 0x7f757c197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f758244e700 1 -- 172.21.15.102:0/2072008860 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757c19f870 con 0x7f757c107f60 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.980+0000 7f7581c4d700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c102d00 0x7f757c197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.981+0000 7f758244e700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c107f60 0x7f757c19f1a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7574007ee0 tx=0x7f757400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.981+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574004db0 con 0x7f757c107f60 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.981+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f7574009e20 con 0x7f757c107f60 2024-03-30T13:53:11.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.981+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757c19fb60 con 0x7f757c107f60 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.981+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574015400 con 0x7f757c107f60 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.981+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f757c1a00a0 con 0x7f757c107f60 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.982+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 13) ==== 46138+0+0 (secure 0 0 0) 0x7f75740155a0 con 0x7f757c107f60 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.982+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f757c068f00 con 0x7f757c107f60 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.982+0000 7f7572ffd700 1 --2- 172.21.15.102:0/2072008860 >> v2:172.21.15.102:6800/3154847229 conn(0x7f75680390a0 0x7f756803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.982+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (secure 0 0 0) 0x7f757404b9c0 con 0x7f757c107f60 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.982+0000 7f7581c4d700 1 --2- 172.21.15.102:0/2072008860 >> v2:172.21.15.102:6800/3154847229 conn(0x7f75680390a0 0x7f756803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:11.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.982+0000 7f7581c4d700 1 --2- 172.21.15.102:0/2072008860 >> v2:172.21.15.102:6800/3154847229 conn(0x7f75680390a0 0x7f756803b560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f757c199100 tx=0x7f757801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:11.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:11.987+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7574010070 con 0x7f757c107f60 2024-03-30T13:53:12.185 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:11.918+0000 7fbfa3ab8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:53:12.185 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:12.186+0000 7fbfa3ab8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:12.196+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f757c02df10 con 0x7f75680390a0 2024-03-30T13:53:12.437 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:12.424+0000 7fbfa3ab8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:12.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:12.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:12.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:12.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:12.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:13.437 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:13.220+0000 7fbfa3ab8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:53:13.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:13 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:53:13.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:13 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:53:13.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:13 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:13.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:13 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:53:13.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:13 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:53:13.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:13 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:13 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:53:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:13 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:53:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:13 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:14.063 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:13.787+0000 7fbfa3ab8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:53:14.063 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:13.883+0000 7fbfa3ab8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:53:14.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:14 smithi138 ceph-mon[33397]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:14.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:14 smithi138 ceph-mon[33397]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:14.438 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:53:14 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:53:14.098+0000 7fbfa3ab8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:53:14.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:14 smithi102 ceph-mon[29392]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:14.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:14 smithi102 ceph-mon[29392]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:14.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:14 smithi102 ceph-mon[37717]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:14.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:14 smithi102 ceph-mon[37717]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:15.073+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mgrmap(e 14) ==== 91684+0+0 (secure 0 0 0) 0x7f7574015c10 con 0x7f757c107f60 2024-03-30T13:53:15.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: Standby manager daemon x started 2024-03-30T13:53:15.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:53:15.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:53:15.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:53:15.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:53:15.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1892558231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]: dispatch 2024-03-30T13:53:15.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]: dispatch 2024-03-30T13:53:15.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]': finished 2024-03-30T13:53:15.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T13:53:15.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:15 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: Standby manager daemon x started 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1892558231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]: dispatch 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]: dispatch 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]': finished 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T13:53:15.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: Standby manager daemon x started 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/4104446314' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1892558231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]: dispatch 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]: dispatch 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9e04b0b-a4df-4d4f-87f8-59e29edd2e82"}]': finished 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T13:53:15.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:15 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:16.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:16 smithi138 ceph-mon[33397]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:16.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:16 smithi138 ceph-mon[33397]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-30T13:53:16.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:53:16.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1494704215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[29392]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[29392]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1494704215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[37717]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[37717]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:53:16.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1494704215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:53:18.289 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:18 smithi102 ceph-mon[29392]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:18.289 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:18 smithi102 ceph-mon[37717]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:18.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:18 smithi138 ceph-mon[33397]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:20.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:20 smithi138 ceph-mon[33397]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:20.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:20 smithi102 ceph-mon[29392]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:20.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:20 smithi102 ceph-mon[37717]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:22.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[37717]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:22.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T13:53:22.372 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:22.373 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[37717]: Deploying daemon osd.0 on smithi102 2024-03-30T13:53:22.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[29392]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:22.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T13:53:22.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:22.373 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:22 smithi102 ceph-mon[29392]: Deploying daemon osd.0 on smithi102 2024-03-30T13:53:22.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:22 smithi138 ceph-mon[33397]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:22.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:22 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T13:53:22.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:22 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:22.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:22 smithi138 ceph-mon[33397]: Deploying daemon osd.0 on smithi102 2024-03-30T13:53:24.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:24 smithi102 ceph-mon[29392]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:24.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:24 smithi102 ceph-mon[37717]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:24.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:24 smithi138 ceph-mon[33397]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:26.351 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:26 smithi102 ceph-mon[37717]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:26.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:26 smithi102 ceph-mon[29392]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:26 smithi138 ceph-mon[33397]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:27.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:27.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:27.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:27.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[29392]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:27.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:27.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:27.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:27.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:27 smithi102 ceph-mon[37717]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:27.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:27.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:27.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:27.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:27 smithi138 ceph-mon[33397]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.796+0000 7f7572ffd700 1 -- 172.21.15.102:0/2072008860 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f757c02df10 con 0x7f75680390a0 2024-03-30T13:53:29.796 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 0 on host 'smithi102' 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 >> v2:172.21.15.102:6800/3154847229 conn(0x7f75680390a0 msgr2=0x7f756803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2072008860 >> v2:172.21.15.102:6800/3154847229 conn(0x7f75680390a0 0x7f756803b560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f757c199100 tx=0x7f757801a040 comp rx=0 tx=0).stop 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c107f60 msgr2=0x7f757c19f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c107f60 0x7f757c19f1a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7574007ee0 tx=0x7f757400d3b0 comp rx=0 tx=0).stop 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 shutdown_connections 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2072008860 >> v2:172.21.15.102:6800/3154847229 conn(0x7f75680390a0 0x7f756803b560 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f757c102d00 0x7f757c197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f757c107f60 0x7f757c19f1a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 --2- 172.21.15.102:0/2072008860 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f757c105630 0x7f757c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:29.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.798+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 >> 172.21.15.102:0/2072008860 conn(0x7f757c0fc710 msgr2=0x7f757c103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.799+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 shutdown_connections 2024-03-30T13:53:29.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:29.799+0000 7f7583eb1700 1 -- 172.21.15.102:0/2072008860 wait complete. 2024-03-30T13:53:30.349 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[29392]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:30.349 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[29392]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:53:30.349 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:30.349 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:30.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[37717]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:30.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[37717]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:53:30.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:30.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:30.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:30 smithi138 ceph-mon[33397]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:30.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:30 smithi138 ceph-mon[33397]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T13:53:30.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:30.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:30.609 DEBUG:teuthology.orchestra.run.smithi102:osd.0> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T13:53:30.611 INFO:tasks.cephadm:Deploying osd.1 on smithi102 with /dev/vg_nvme/lv_3... 2024-03-30T13:53:30.611 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T13:53:30.895 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:31.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[37717]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T13:53:31.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[37717]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[37717]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[37717]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[29392]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[29392]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[29392]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:31.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:31.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-mon[29392]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:53:31.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:31 smithi138 ceph-mon[33397]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T13:53:31.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:31 smithi138 ceph-mon[33397]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T13:53:31.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:31 smithi138 ceph-mon[33397]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:31.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:31 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:31.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:31 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:31.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:31 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:31.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:31 smithi138 ceph-mon[33397]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:53:31.691 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0[41735]: 2024-03-30T13:53:31.485+0000 7faa469a8700 -1 osd.0 0 waiting for initial osdmap 2024-03-30T13:53:31.691 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:53:31 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0[41735]: 2024-03-30T13:53:31.491+0000 7faa3e793700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:53:32.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[37717]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:32.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[37717]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T13:53:32.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[37717]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' 2024-03-30T13:53:32.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[29392]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:32.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[29392]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T13:53:32.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[29392]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' 2024-03-30T13:53:32.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:32 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:32 smithi138 ceph-mon[33397]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:32.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:32 smithi138 ceph-mon[33397]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T13:53:32.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:32 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:32 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:32 smithi138 ceph-mon[33397]: from='osd.0 v2:172.21.15.102:6801/3100565996' entity='osd.0' 2024-03-30T13:53:32.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:32 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:32.818 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T13:53:32.818 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T13:53:32.818 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-30T13:53:32.818 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-30T13:53:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140776 s, 745 MB/s 2024-03-30T13:53:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-30T13:53:33.726 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:53:33.726 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:53:33.726 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[29392]: osd.0 v2:172.21.15.102:6801/3100565996 boot 2024-03-30T13:53:33.726 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[29392]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[29392]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[37717]: osd.0 v2:172.21.15.102:6801/3100565996 boot 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[37717]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:33.727 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:33 smithi102 ceph-mon[37717]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:33.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:33 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:53:33.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:33 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:53:33.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:33 smithi138 ceph-mon[33397]: osd.0 v2:172.21.15.102:6801/3100565996 boot 2024-03-30T13:53:33.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:33 smithi138 ceph-mon[33397]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T13:53:33.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:33 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:53:33.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:33 smithi138 ceph-mon[33397]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T13:53:34.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2024-03-30T13:53:34.665 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:34.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:34 smithi102 ceph-mon[29392]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T13:53:34.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:34 smithi102 ceph-mon[37717]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T13:53:34.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:34 smithi138 ceph-mon[33397]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T13:53:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:35 smithi102 ceph-mon[29392]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:35.675 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:35 smithi102 ceph-mon[37717]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:35.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:35 smithi138 ceph-mon[33397]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 -- 172.21.15.102:0/138453851 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 msgr2=0x7f971010a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 --2- 172.21.15.102:0/138453851 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971010a6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9708009f90 tx=0x7f970800b330 comp rx=0 tx=0).stop 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 -- 172.21.15.102:0/138453851 shutdown_connections 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 --2- 172.21.15.102:0/138453851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710102270 0x7f9710102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 --2- 172.21.15.102:0/138453851 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971010a6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 --2- 172.21.15.102:0/138453851 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9710103470 0x7f97101038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:36.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 -- 172.21.15.102:0/138453851 >> 172.21.15.102:0/138453851 conn(0x7f97100fd850 msgr2=0x7f97100ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:36.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 -- 172.21.15.102:0/138453851 shutdown_connections 2024-03-30T13:53:36.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 -- 172.21.15.102:0/138453851 wait complete. 2024-03-30T13:53:36.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 Processor -- start 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.886+0000 7f971865b700 1 -- start start 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f971865b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9710102270 0x7f9710197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f971865b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710103470 0x7f9710198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f971865b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f971865b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f971010d5a0 con 0x7f9710103470 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f971865b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f971010d420 con 0x7f9710102270 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f971865b700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f971010d720 con 0x7f9710103e10 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710103470 0x7f9710198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9716bf8700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9716bf8700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:42368/0 (socket says 172.21.15.102:42368) 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710103470 0x7f9710198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52490/0 (socket says 172.21.15.102:52490) 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9716bf8700 1 -- 172.21.15.102:0/1094664081 learned_addr learned my addr 172.21.15.102:0/1094664081 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:53:36.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f97163f7700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9710102270 0x7f9710197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 -- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 msgr2=0x7f971019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 -- 172.21.15.102:0/1094664081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9710102270 msgr2=0x7f9710197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9710102270 0x7f9710197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 -- 172.21.15.102:0/1094664081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971019f750 con 0x7f9710103470 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9716bf8700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f9715bf6700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710103470 0x7f9710198030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f970c00ea20 tx=0x7f970c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f97077fe700 1 -- 172.21.15.102:0/1094664081 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970c00cbd0 con 0x7f9710103470 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f971019fa40 con 0x7f9710103470 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f97077fe700 1 -- 172.21.15.102:0/1094664081 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f970c00ee60 con 0x7f9710103470 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.887+0000 7f97077fe700 1 -- 172.21.15.102:0/1094664081 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970c005020 con 0x7f9710103470 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.888+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f971019ff80 con 0x7f9710103470 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.888+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9710068f00 con 0x7f9710103470 2024-03-30T13:53:36.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.889+0000 7f97077fe700 1 -- 172.21.15.102:0/1094664081 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 14) ==== 91684+0+0 (secure 0 0 0) 0x7f970c0051c0 con 0x7f9710103470 2024-03-30T13:53:36.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.889+0000 7f97077fe700 1 --2- 172.21.15.102:0/1094664081 >> v2:172.21.15.102:6800/3154847229 conn(0x7f96fc06da80 0x7f96fc06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:53:36.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.889+0000 7f97077fe700 1 -- 172.21.15.102:0/1094664081 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1541+0+0 (secure 0 0 0) 0x7f970c08c510 con 0x7f9710103470 2024-03-30T13:53:36.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.889+0000 7f97163f7700 1 --2- 172.21.15.102:0/1094664081 >> v2:172.21.15.102:6800/3154847229 conn(0x7f96fc06da80 0x7f96fc06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:53:36.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.890+0000 7f97163f7700 1 --2- 172.21.15.102:0/1094664081 >> v2:172.21.15.102:6800/3154847229 conn(0x7f96fc06da80 0x7f96fc06ff40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9700009750 tx=0x7f9700009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:53:36.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:36.894+0000 7f97077fe700 1 -- 172.21.15.102:0/1094664081 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f970c05b340 con 0x7f9710103470 2024-03-30T13:53:37.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:37.101+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f971002ddf0 con 0x7f96fc06da80 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi102 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi102 to 16064M 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:37.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi102 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.738 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi102 to 16064M 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:37.739 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi102 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi102 to 16064M 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:37.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:37.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:40.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:40 smithi138 ceph-mon[33397]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:40.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1645928662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]: dispatch 2024-03-30T13:53:40.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]: dispatch 2024-03-30T13:53:40.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]': finished 2024-03-30T13:53:40.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:40 smithi138 ceph-mon[33397]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T13:53:40.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:40 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[37717]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1645928662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]: dispatch 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]: dispatch 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]': finished 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[37717]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[29392]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1645928662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]: dispatch 2024-03-30T13:53:40.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]: dispatch 2024-03-30T13:53:40.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "046093f3-4186-496a-a5b2-9e8d4b86a2ad"}]': finished 2024-03-30T13:53:40.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[29392]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T13:53:40.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:40 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:41.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1275284630' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:53:41.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1275284630' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:53:41.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1275284630' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:53:42.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:42 smithi138 ceph-mon[33397]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:42.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:42 smithi102 ceph-mon[29392]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:42.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:42 smithi102 ceph-mon[37717]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:44.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:44 smithi102 ceph-mon[29392]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:44.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:44 smithi102 ceph-mon[37717]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:44.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:44 smithi138 ceph-mon[33397]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:46.286 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:46 smithi102 ceph-mon[37717]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:46.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:46 smithi102 ceph-mon[29392]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:46.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:46 smithi138 ceph-mon[33397]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:47.180 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:47 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T13:53:47.180 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:47 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:47.181 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T13:53:47.181 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:47 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:47.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:47 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T13:53:47.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:47 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:48.155 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:48 smithi102 ceph-mon[29392]: Deploying daemon osd.1 on smithi102 2024-03-30T13:53:48.155 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:48 smithi102 ceph-mon[29392]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:48.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:48 smithi102 ceph-mon[37717]: Deploying daemon osd.1 on smithi102 2024-03-30T13:53:48.155 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:48 smithi102 ceph-mon[37717]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:48.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:48 smithi138 ceph-mon[33397]: Deploying daemon osd.1 on smithi102 2024-03-30T13:53:48.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:48 smithi138 ceph-mon[33397]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:50.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:50 smithi102 ceph-mon[29392]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:50.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:50 smithi102 ceph-mon[37717]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:50.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:50 smithi138 ceph-mon[33397]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[37717]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[29392]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:52.406 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:52.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:52 smithi138 ceph-mon[33397]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:52.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:52.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:52.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:54.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:54 smithi102 ceph-mon[29392]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:54.367 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:54 smithi102 ceph-mon[37717]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:54.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:54 smithi138 ceph-mon[33397]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:55.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.410+0000 7f97077fe700 1 -- 172.21.15.102:0/1094664081 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f971002ddf0 con 0x7f96fc06da80 2024-03-30T13:53:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 1 on host 'smithi102' 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 >> v2:172.21.15.102:6800/3154847229 conn(0x7f96fc06da80 msgr2=0x7f96fc06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 --2- 172.21.15.102:0/1094664081 >> v2:172.21.15.102:6800/3154847229 conn(0x7f96fc06da80 0x7f96fc06ff40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9700009750 tx=0x7f9700009310 comp rx=0 tx=0).stop 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710103470 msgr2=0x7f9710198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710103470 0x7f9710198030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f970c00ea20 tx=0x7f970c00ed30 comp rx=0 tx=0).stop 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 shutdown_connections 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 --2- 172.21.15.102:0/1094664081 >> v2:172.21.15.102:6800/3154847229 conn(0x7f96fc06da80 0x7f96fc06ff40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9710103e10 0x7f971019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9710102270 0x7f9710197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 --2- 172.21.15.102:0/1094664081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9710103470 0x7f9710198030 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:53:55.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 >> 172.21.15.102:0/1094664081 conn(0x7f97100fd850 msgr2=0x7f9710106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:53:55.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 shutdown_connections 2024-03-30T13:53:55.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:53:55.412+0000 7f971865b700 1 -- 172.21.15.102:0/1094664081 wait complete. 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='osd.1 v2:172.21.15.102:6805/3593563159' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='osd.1 v2:172.21.15.102:6805/3593563159' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:55 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='osd.1 v2:172.21.15.102:6805/3593563159' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:53:55.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:55 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:53:56.151 DEBUG:teuthology.orchestra.run.smithi102:osd.1> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-30T13:53:56.154 INFO:tasks.cephadm:Deploying osd.2 on smithi102 with /dev/vg_nvme/lv_2... 2024-03-30T13:53:56.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T13:53:56.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:56 smithi138 ceph-mon[33397]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T13:53:56.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:56 smithi138 ceph-mon[33397]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T13:53:56.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:56 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:56.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:56 smithi138 ceph-mon[33397]: from='osd.1 v2:172.21.15.102:6805/3593563159' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:56.938 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:56 smithi138 ceph-mon[33397]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:56.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[37717]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T13:53:56.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[37717]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[37717]: from='osd.1 v2:172.21.15.102:6805/3593563159' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[37717]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[29392]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[29392]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[29392]: from='osd.1 v2:172.21.15.102:6805/3593563159' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:56.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:56 smithi102 ceph-mon[29392]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:53:57.278 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:53:57.370 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1[47444]: 2024-03-30T13:53:57.039+0000 7f0e58f96700 -1 osd.1 0 waiting for initial osdmap 2024-03-30T13:53:57.370 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1[47444]: 2024-03-30T13:53:57.042+0000 7f0e50d81700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:53:57.621 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[37717]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:53:57.621 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[37717]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T13:53:57.621 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:57.621 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:57.621 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[37717]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:57.621 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[37717]: from='osd.1 ' entity='osd.1' 2024-03-30T13:53:57.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[29392]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:53:57.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[29392]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T13:53:57.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:57.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:57.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[29392]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:57.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:57 smithi102 ceph-mon[29392]: from='osd.1 ' entity='osd.1' 2024-03-30T13:53:57.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:57 smithi138 ceph-mon[33397]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:53:57.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:57 smithi138 ceph-mon[33397]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T13:53:57.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:57.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:57 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:57.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:57 smithi138 ceph-mon[33397]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T13:53:57.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:57 smithi138 ceph-mon[33397]: from='osd.1 ' entity='osd.1' 2024-03-30T13:53:58.909 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[29392]: osd.1 v2:172.21.15.102:6805/3593563159 boot 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[29392]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[37717]: osd.1 v2:172.21.15.102:6805/3593563159 boot 2024-03-30T13:53:58.910 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[37717]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T13:53:58.911 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:58.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:58 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:53:58.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:58 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:53:58.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:58.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:58 smithi138 ceph-mon[33397]: osd.1 v2:172.21.15.102:6805/3593563159 boot 2024-03-30T13:53:58.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:58 smithi138 ceph-mon[33397]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T13:53:58.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:53:59.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:53:59 smithi102 ceph-mon[29392]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:53:59.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:53:59 smithi102 ceph-mon[37717]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:53:59.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:53:59 smithi138 ceph-mon[33397]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T13:54:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T13:54:00.357 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-30T13:54:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-30T13:54:00.357 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121423 s, 864 MB/s 2024-03-30T13:54:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi102 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi102 to 8032M 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:00.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T13:54:00.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi102 2024-03-30T13:54:00.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi102 to 8032M 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi102 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:00.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:00.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi102 to 8032M 2024-03-30T13:54:00.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:00.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:00.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:00.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2024-03-30T13:54:01.224 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:01.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:01 smithi138 ceph-mon[33397]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:01.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:01 smithi102 ceph-mon[29392]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:01.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:01 smithi102 ceph-mon[37717]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:02.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.750+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3172690949 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81011d0 msgr2=0x7fb8c81015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.750+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3172690949 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81011d0 0x7fb8c81015f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009920 tx=0x7fb8c4009c30 comp rx=0 tx=0).stop 2024-03-30T13:54:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.750+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3172690949 shutdown_connections 2024-03-30T13:54:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.750+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3172690949 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81011d0 0x7fb8c81015f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.750+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3172690949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8c8102d70 0x7fb8c8109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.750+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3172690949 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c81023d0 0x7fb8c8102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.750+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3172690949 >> 172.21.15.102:0/3172690949 conn(0x7fb8c80fc790 msgr2=0x7fb8c80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3172690949 shutdown_connections 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3172690949 wait complete. 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 Processor -- start 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 -- start start 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8c81011d0 0x7fb8c8197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81023d0 0x7fb8c81981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c8102d70 0x7fb8c819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c810c4d0 con 0x7fb8c81023d0 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c810c350 con 0x7fb8c81011d0 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8cfd6e700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c810c650 con 0x7fb8c8102d70 2024-03-30T13:54:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.751+0000 7fb8ce30b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c8102d70 0x7fb8c819f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81023d0 0x7fb8c81981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81023d0 0x7fb8c81981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36050/0 (socket says 172.21.15.102:36050) 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 -- 172.21.15.102:0/3553496132 learned_addr learned my addr 172.21.15.102:0/3553496132 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8ce30b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c8102d70 0x7fb8c819f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:59032/0 (socket says 172.21.15.102:59032) 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cdb0a700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8c81011d0 0x7fb8c8197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 -- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c8102d70 msgr2=0x7fb8c819f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c8102d70 0x7fb8c819f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 -- 172.21.15.102:0/3553496132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8c81011d0 msgr2=0x7fb8c8197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8c81011d0 0x7fb8c8197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 -- 172.21.15.102:0/3553496132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c819f920 con 0x7fb8c81023d0 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8ce30b700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c8102d70 0x7fb8c819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cd309700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81023d0 0x7fb8c81981e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800ec10 tx=0x7fb8b800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8beffd700 1 -- 172.21.15.102:0/3553496132 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b800cc90 con 0x7fb8c81023d0 2024-03-30T13:54:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c819fc10 con 0x7fb8c81023d0 2024-03-30T13:54:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8beffd700 1 -- 172.21.15.102:0/3553496132 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fb8b800ce30 con 0x7fb8c81023d0 2024-03-30T13:54:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8beffd700 1 -- 172.21.15.102:0/3553496132 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8010430 con 0x7fb8c81023d0 2024-03-30T13:54:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.752+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c81a0150 con 0x7fb8c81023d0 2024-03-30T13:54:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.753+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8c8068f00 con 0x7fb8c81023d0 2024-03-30T13:54:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.754+0000 7fb8beffd700 1 -- 172.21.15.102:0/3553496132 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 14) ==== 91684+0+0 (secure 0 0 0) 0x7fb8b8003fc0 con 0x7fb8c81023d0 2024-03-30T13:54:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.755+0000 7fb8beffd700 1 --2- 172.21.15.102:0/3553496132 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb8b406da80 0x7fb8b406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.755+0000 7fb8beffd700 1 -- 172.21.15.102:0/3553496132 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 1833+0+0 (secure 0 0 0) 0x7fb8b808ca40 con 0x7fb8c81023d0 2024-03-30T13:54:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.755+0000 7fb8cdb0a700 1 --2- 172.21.15.102:0/3553496132 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb8b406da80 0x7fb8b406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.755+0000 7fb8cdb0a700 1 --2- 172.21.15.102:0/3553496132 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb8b406da80 0x7fb8b406ff40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40098f0 tx=0x7fb8c4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:54:02.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.763+0000 7fb8beffd700 1 -- 172.21.15.102:0/3553496132 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb8b805b750 con 0x7fb8c81023d0 2024-03-30T13:54:02.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:02.971+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb8c8062680 con 0x7fb8b406da80 2024-03-30T13:54:03.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:03.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:03.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:03.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:03.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:03.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:03.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:03.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:03.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:04.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:04 smithi102 ceph-mon[37717]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:04.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:04 smithi102 ceph-mon[37717]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:04.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:04 smithi102 ceph-mon[29392]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:04.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:04 smithi102 ceph-mon[29392]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:04.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:04 smithi138 ceph-mon[33397]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:04.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:04 smithi138 ceph-mon[33397]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:06.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[37717]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2707424792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4779686a-2976-4755-9116-8835f28fa5a9"}]: dispatch 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2707424792' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4779686a-2976-4755-9116-8835f28fa5a9"}]': finished 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[37717]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[29392]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2707424792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4779686a-2976-4755-9116-8835f28fa5a9"}]: dispatch 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2707424792' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4779686a-2976-4755-9116-8835f28fa5a9"}]': finished 2024-03-30T13:54:06.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[29392]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T13:54:06.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:06 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:06.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:06 smithi138 ceph-mon[33397]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:06.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2707424792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4779686a-2976-4755-9116-8835f28fa5a9"}]: dispatch 2024-03-30T13:54:06.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2707424792' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4779686a-2976-4755-9116-8835f28fa5a9"}]': finished 2024-03-30T13:54:06.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:06 smithi138 ceph-mon[33397]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T13:54:06.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:06 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:07.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2599092691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:07.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2599092691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:07.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2599092691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:08.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:08 smithi138 ceph-mon[33397]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:08.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:08 smithi102 ceph-mon[29392]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:08.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:08 smithi102 ceph-mon[37717]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:10.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:10 smithi138 ceph-mon[33397]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:10.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:10 smithi102 ceph-mon[29392]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:10.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:10 smithi102 ceph-mon[37717]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:12.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:12 smithi102 ceph-mon[29392]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:12.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:12 smithi102 ceph-mon[37717]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:12.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:12 smithi138 ceph-mon[33397]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:13.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:13 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T13:54:13.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:13 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:13.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:13 smithi138 ceph-mon[33397]: Deploying daemon osd.2 on smithi102 2024-03-30T13:54:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:13 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T13:54:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:13 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:13 smithi102 ceph-mon[29392]: Deploying daemon osd.2 on smithi102 2024-03-30T13:54:13.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:13 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T13:54:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:13 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:13 smithi102 ceph-mon[37717]: Deploying daemon osd.2 on smithi102 2024-03-30T13:54:14.292 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:14 smithi102 ceph-mon[29392]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:14.292 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:14 smithi102 ceph-mon[37717]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:14.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:14 smithi138 ceph-mon[33397]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:16.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:16 smithi138 ceph-mon[33397]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:16.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:16 smithi102 ceph-mon[29392]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:16.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:16 smithi102 ceph-mon[37717]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[29392]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[37717]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:18.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:18.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:18 smithi138 ceph-mon[33397]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:18.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:18.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:18.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:20.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:20 smithi102 ceph-mon[29392]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:20.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:20 smithi102 ceph-mon[37717]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:20.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:20 smithi138 ceph-mon[33397]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 2 on host 'smithi102' 2024-03-30T13:54:21.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.440+0000 7fb8beffd700 1 -- 172.21.15.102:0/3553496132 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb8c8062680 con 0x7fb8b406da80 2024-03-30T13:54:21.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb8b406da80 msgr2=0x7fb8b406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:21.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3553496132 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb8b406da80 0x7fb8b406ff40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40098f0 tx=0x7fb8c4019040 comp rx=0 tx=0).stop 2024-03-30T13:54:21.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81023d0 msgr2=0x7fb8c81981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:21.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81023d0 0x7fb8c81981e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800ec10 tx=0x7fb8b800ef20 comp rx=0 tx=0).stop 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 shutdown_connections 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3553496132 >> v2:172.21.15.102:6800/3154847229 conn(0x7fb8b406da80 0x7fb8b406ff40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb8c8102d70 0x7fb8c819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8c81011d0 0x7fb8c8197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 --2- 172.21.15.102:0/3553496132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8c81023d0 0x7fb8c81981e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 >> 172.21.15.102:0/3553496132 conn(0x7fb8c80fc790 msgr2=0x7fb8c80fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 shutdown_connections 2024-03-30T13:54:21.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:21.443+0000 7fb8cfd6e700 1 -- 172.21.15.102:0/3553496132 wait complete. 2024-03-30T13:54:21.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='osd.2 v2:172.21.15.102:6809/203063794' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='osd.2 v2:172.21.15.102:6809/203063794' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:21 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='osd.2 v2:172.21.15.102:6809/203063794' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:21.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:21 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:22.133 DEBUG:teuthology.orchestra.run.smithi102:osd.2> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T13:54:22.136 INFO:tasks.cephadm:Deploying osd.3 on smithi102 with /dev/vg_nvme/lv_1... 2024-03-30T13:54:22.136 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T13:54:22.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:22 smithi138 ceph-mon[33397]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T13:54:22.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:22 smithi138 ceph-mon[33397]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T13:54:22.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:22 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:22.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:22 smithi138 ceph-mon[33397]: from='osd.2 v2:172.21.15.102:6809/203063794' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:22.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:22 smithi138 ceph-mon[33397]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[29392]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T13:54:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[29392]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T13:54:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[29392]: from='osd.2 v2:172.21.15.102:6809/203063794' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[29392]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:22.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[37717]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T13:54:22.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[37717]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T13:54:22.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:22.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[37717]: from='osd.2 v2:172.21.15.102:6809/203063794' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:22.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:22 smithi102 ceph-mon[37717]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:23.190 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2[53048]: 2024-03-30T13:54:23.112+0000 7fb3dafb3700 -1 osd.2 0 waiting for initial osdmap 2024-03-30T13:54:23.190 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2[53048]: 2024-03-30T13:54:23.117+0000 7fb3d2d9e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:54:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:23.859 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[37717]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:54:23.859 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[37717]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T13:54:23.859 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[37717]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[37717]: from='osd.2 ' entity='osd.2' 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[29392]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[29392]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[29392]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:23.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:23 smithi102 ceph-mon[29392]: from='osd.2 ' entity='osd.2' 2024-03-30T13:54:23.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:23 smithi138 ceph-mon[33397]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:54:23.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:23 smithi138 ceph-mon[33397]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T13:54:23.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:23 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:23.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:23 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:23.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:23 smithi138 ceph-mon[33397]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-30T13:54:23.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:23 smithi138 ceph-mon[33397]: from='osd.2 ' entity='osd.2' 2024-03-30T13:54:24.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:24 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:54:24.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:24 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:54:24.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:24 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:24.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:24 smithi138 ceph-mon[33397]: osd.2 v2:172.21.15.102:6809/203063794 boot 2024-03-30T13:54:24.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:24 smithi138 ceph-mon[33397]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T13:54:24.937 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:24 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[29392]: osd.2 v2:172.21.15.102:6809/203063794 boot 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[29392]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:54:24.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:54:24.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:24.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[37717]: osd.2 v2:172.21.15.102:6809/203063794 boot 2024-03-30T13:54:24.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[37717]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T13:54:24.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:24 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113602 s, 923 MB/s 2024-03-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi102 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:26.392 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi102 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:26.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:26.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T13:54:26.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T13:54:26.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:54:26.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi102 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T13:54:26.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:54:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:27 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi102 to 5354M 2024-03-30T13:54:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:54:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:27 smithi102 ceph-mon[29392]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T13:54:27.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:27 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi102 to 5354M 2024-03-30T13:54:27.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:54:27.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:27 smithi102 ceph-mon[37717]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T13:54:27.137 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57236]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:54:27.137 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57236]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:27.137 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57236]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:27.138 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57236]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:27.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi102:vg_nvme/lv_1 2024-03-30T13:54:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:54:27.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:27 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi102 to 5354M 2024-03-30T13:54:27.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T13:54:27.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:27 smithi138 ceph-mon[33397]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T13:54:27.440 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57257]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:54:27.440 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57257]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:27.440 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57257]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:27.440 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57257]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:27.783 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57316]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:54:27.783 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57316]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:27.783 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57316]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:27.783 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57316]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:28.067 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57337]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:54:28.067 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57337]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:28.067 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57337]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:28.067 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 13:54:27 smithi102 sudo[57337]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:28.329 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57343]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:54:28.329 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:28.329 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57343]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:28.329 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57343]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:28.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[29392]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:28.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:28.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[29392]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T13:54:28.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[29392]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T13:54:28.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[37717]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:28.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:28.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[37717]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T13:54:28.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:28 smithi102 ceph-mon[37717]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T13:54:28.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:28 smithi138 ceph-mon[33397]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:28.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:28 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:28.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:28 smithi138 ceph-mon[33397]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T13:54:28.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:28 smithi138 ceph-mon[33397]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T13:54:28.588 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57346]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:54:28.589 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:28.589 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57346]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:28.589 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57346]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:28.859 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57369]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:54:28.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57369]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:28.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57369]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:28.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57369]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:29.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.011+0000 7fdea8ef9700 1 -- 172.21.15.102:0/512527782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4102370 msgr2=0x7fdea41027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:29.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.011+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/512527782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4102370 0x7fdea41027d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde98009990 tx=0x7fde98009ca0 comp rx=0 tx=0).stop 2024-03-30T13:54:29.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 -- 172.21.15.102:0/512527782 shutdown_connections 2024-03-30T13:54:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/512527782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdea4101170 0x7fdea4101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/512527782 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdea4102d10 0x7fdea41095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/512527782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4102370 0x7fdea41027d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 -- 172.21.15.102:0/512527782 >> 172.21.15.102:0/512527782 conn(0x7fdea40fc730 msgr2=0x7fdea40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:54:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 -- 172.21.15.102:0/512527782 shutdown_connections 2024-03-30T13:54:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 -- 172.21.15.102:0/512527782 wait complete. 2024-03-30T13:54:29.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 Processor -- start 2024-03-30T13:54:29.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.012+0000 7fdea8ef9700 1 -- start start 2024-03-30T13:54:29.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea8ef9700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4101170 0x7fdea406b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:29.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea8ef9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdea4102370 0x7fdea419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:29.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea8ef9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdea4102d10 0x7fdea419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:29.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea8ef9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdea410c480 con 0x7fdea4102d10 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea8ef9700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdea410c300 con 0x7fdea4102370 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea8ef9700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdea410c600 con 0x7fdea4101170 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea259c700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4101170 0x7fdea406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdea4102d10 0x7fdea419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea259c700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4101170 0x7fdea406b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:36048/0 (socket says 172.21.15.102:36048) 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea259c700 1 -- 172.21.15.102:0/3367745554 learned_addr learned my addr 172.21.15.102:0/3367745554 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea1d9b700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdea4102370 0x7fdea419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea2d9d700 1 -- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4101170 msgr2=0x7fdea406b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea2d9d700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4101170 0x7fdea406b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea2d9d700 1 -- 172.21.15.102:0/3367745554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdea4102370 msgr2=0x7fdea419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea2d9d700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdea4102370 0x7fdea419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea2d9d700 1 -- 172.21.15.102:0/3367745554 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea419f780 con 0x7fdea4102d10 2024-03-30T13:54:29.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea259c700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4101170 0x7fdea406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:54:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea2d9d700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdea4102d10 0x7fdea419eeb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde9c007f10 tx=0x7fde9c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:54:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fde937fe700 1 -- 172.21.15.102:0/3367745554 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c004db0 con 0x7fdea4102d10 2024-03-30T13:54:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fde937fe700 1 -- 172.21.15.102:0/3367745554 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fde9c009e20 con 0x7fdea4102d10 2024-03-30T13:54:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea419fa70 con 0x7fdea4102d10 2024-03-30T13:54:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.013+0000 7fde937fe700 1 -- 172.21.15.102:0/3367745554 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c015400 con 0x7fdea4102d10 2024-03-30T13:54:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.014+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea419ff60 con 0x7fdea4102d10 2024-03-30T13:54:29.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.014+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea4068f00 con 0x7fdea4102d10 2024-03-30T13:54:29.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.015+0000 7fde937fe700 1 -- 172.21.15.102:0/3367745554 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7fde9c00db20 con 0x7fdea4102d10 2024-03-30T13:54:29.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.015+0000 7fde937fe700 1 --2- 172.21.15.102:0/3367745554 >> v2:172.21.15.102:6800/3154847229 conn(0x7fde8c06db10 0x7fde8c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:29.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.016+0000 7fde937fe700 1 -- 172.21.15.102:0/3367745554 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 2511+0+0 (secure 0 0 0) 0x7fde9c08c4c0 con 0x7fdea4102d10 2024-03-30T13:54:29.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.016+0000 7fdea259c700 1 --2- 172.21.15.102:0/3367745554 >> v2:172.21.15.102:6800/3154847229 conn(0x7fde8c06db10 0x7fde8c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:29.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.016+0000 7fdea259c700 1 --2- 172.21.15.102:0/3367745554 >> v2:172.21.15.102:6800/3154847229 conn(0x7fde8c06db10 0x7fde8c06ffd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde94005950 tx=0x7fde94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:54:29.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.021+0000 7fde937fe700 1 -- 172.21.15.102:0/3367745554 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fde9c010070 con 0x7fdea4102d10 2024-03-30T13:54:29.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57377]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:54:29.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57377]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:29.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57377]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:29.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:28 smithi102 sudo[57377]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:29.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:29.226+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdea402df10 con 0x7fde8c06db10 2024-03-30T13:54:29.357 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35972]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:54:29.357 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35972]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:29.358 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35972]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:29.358 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35972]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:29.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:29.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:29.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:29.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:29.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:29.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:29.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:29.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35975]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35975]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35975]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 sudo[35975]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:29.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:29 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:29.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57469]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T13:54:29.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57469]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:29.898 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57469]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:29.899 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57469]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:30.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T13:54:30.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T13:54:30.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57529]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T13:54:30.169 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:29 smithi102 sudo[57529]: pam_unix(sudo:session): session closed for user root 2024-03-30T13:54:30.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:30.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:30.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[29392]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:30.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:30.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:30 smithi102 ceph-mon[37717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:30.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:30.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:54:30.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:30 smithi138 ceph-mon[33397]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T13:54:31.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:31 smithi102 ceph-mon[29392]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:31.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:31 smithi102 ceph-mon[37717]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:31.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:31 smithi138 ceph-mon[33397]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:32.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3540700444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]: dispatch 2024-03-30T13:54:32.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]: dispatch 2024-03-30T13:54:32.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]': finished 2024-03-30T13:54:32.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:32 smithi138 ceph-mon[33397]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T13:54:32.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:32 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:32.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3540700444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]: dispatch 2024-03-30T13:54:32.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]: dispatch 2024-03-30T13:54:32.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]': finished 2024-03-30T13:54:32.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[29392]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T13:54:32.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:32.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3540700444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]: dispatch 2024-03-30T13:54:32.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]: dispatch 2024-03-30T13:54:32.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2cd657e0-4c72-45db-a26b-d86c06d2996b"}]': finished 2024-03-30T13:54:32.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[37717]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T13:54:32.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:32 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:33.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/887148959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:33.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:33 smithi138 ceph-mon[33397]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:33.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/887148959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:33.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:33 smithi102 ceph-mon[29392]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:33.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/887148959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:33.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:33 smithi102 ceph-mon[37717]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:36.409 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:36 smithi102 ceph-mon[29392]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T13:54:36.410 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:36 smithi102 ceph-mon[37717]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T13:54:36.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:36 smithi138 ceph-mon[33397]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-30T13:54:38.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:38 smithi102 ceph-mon[29392]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T13:54:38.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:38 smithi102 ceph-mon[37717]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T13:54:38.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:38 smithi138 ceph-mon[33397]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-30T13:54:39.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:39 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T13:54:39.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:39 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:39.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:39 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T13:54:39.331 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:39 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:39 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T13:54:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:39 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:40.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:40 smithi102 ceph-mon[29392]: Deploying daemon osd.3 on smithi102 2024-03-30T13:54:40.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:40 smithi102 ceph-mon[29392]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:54:40.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:40 smithi102 ceph-mon[37717]: Deploying daemon osd.3 on smithi102 2024-03-30T13:54:40.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:40 smithi102 ceph-mon[37717]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:54:40.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:40 smithi138 ceph-mon[33397]: Deploying daemon osd.3 on smithi102 2024-03-30T13:54:40.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:40 smithi138 ceph-mon[33397]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:54:42.349 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:42 smithi102 ceph-mon[29392]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:54:42.349 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:42 smithi102 ceph-mon[37717]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:54:42.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:42 smithi138 ceph-mon[33397]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-30T13:54:44.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:44 smithi102 ceph-mon[37717]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-30T13:54:44.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:44 smithi102 ceph-mon[29392]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-30T13:54:44.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:44 smithi138 ceph-mon[33397]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-30T13:54:45.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:45 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:45.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:45 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:45.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:45 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:45.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:45 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:45.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:45 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:45.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:45 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:45 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:45 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:45 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:46 smithi138 ceph-mon[33397]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T13:54:46.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:46 smithi102 ceph-mon[29392]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T13:54:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:46 smithi102 ceph-mon[37717]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-30T13:54:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.980+0000 7fde937fe700 1 -- 172.21.15.102:0/3367745554 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdea402df10 con 0x7fde8c06db10 2024-03-30T13:54:47.978 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 3 on host 'smithi102' 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.982+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 >> v2:172.21.15.102:6800/3154847229 conn(0x7fde8c06db10 msgr2=0x7fde8c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.982+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/3367745554 >> v2:172.21.15.102:6800/3154847229 conn(0x7fde8c06db10 0x7fde8c06ffd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde94005950 tx=0x7fde94009260 comp rx=0 tx=0).stop 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdea4102d10 msgr2=0x7fdea419eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdea4102d10 0x7fdea419eeb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde9c007f10 tx=0x7fde9c00d3b0 comp rx=0 tx=0).stop 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 shutdown_connections 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/3367745554 >> v2:172.21.15.102:6800/3154847229 conn(0x7fde8c06db10 0x7fde8c06ffd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdea4102d10 0x7fdea419eeb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdea4101170 0x7fdea406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 --2- 172.21.15.102:0/3367745554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdea4102370 0x7fdea419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 >> 172.21.15.102:0/3367745554 conn(0x7fdea40fc730 msgr2=0x7fdea4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:54:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 shutdown_connections 2024-03-30T13:54:47.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:54:47.983+0000 7fdea8ef9700 1 -- 172.21.15.102:0/3367745554 wait complete. 2024-03-30T13:54:48.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:48.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:48.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:48.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='osd.3 v2:172.21.15.102:6813/1080330048' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:54:48.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:54:48.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='osd.3 v2:172.21.15.102:6813/1080330048' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='osd.3 v2:172.21.15.102:6813/1080330048' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:48.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:54:48.627 DEBUG:teuthology.orchestra.run.smithi102:osd.3> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-30T13:54:48.630 INFO:tasks.cephadm:Deploying osd.4 on smithi138 with /dev/vg_nvme/lv_4... 2024-03-30T13:54:48.630 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T13:54:48.874 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:54:49.258 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:49 smithi138 ceph-mon[33397]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T13:54:49.258 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:49 smithi138 ceph-mon[33397]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T13:54:49.258 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:49 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:49.258 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:49 smithi138 ceph-mon[33397]: from='osd.3 v2:172.21.15.102:6813/1080330048' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:49.258 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:49 smithi138 ceph-mon[33397]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:49.258 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:49 smithi138 ceph-mon[33397]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[37717]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T13:54:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[37717]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T13:54:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[37717]: from='osd.3 v2:172.21.15.102:6813/1080330048' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[37717]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[37717]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:49.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[29392]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T13:54:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[29392]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T13:54:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[29392]: from='osd.3 v2:172.21.15.102:6813/1080330048' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[29392]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-30T13:54:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-mon[29392]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:49.940 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T13:54:49.654+0000 7f6b716bf700 -1 osd.3 0 waiting for initial osdmap 2024-03-30T13:54:49.940 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 13:54:49 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T13:54:49.657+0000 7f6b694aa700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:54:50.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[29392]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:54:50.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[29392]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T13:54:50.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[29392]: from='osd.3 ' entity='osd.3' 2024-03-30T13:54:50.287 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[37717]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:54:50.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[37717]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T13:54:50.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[37717]: from='osd.3 ' entity='osd.3' 2024-03-30T13:54:50.288 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:50 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:50 smithi138 ceph-mon[33397]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-30T13:54:50.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:50 smithi138 ceph-mon[33397]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T13:54:50.389 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:50 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.389 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:50 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.389 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:50 smithi138 ceph-mon[33397]: from='osd.3 ' entity='osd.3' 2024-03-30T13:54:50.389 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:50 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:50.961 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T13:54:50.961 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T13:54:50.961 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-30T13:54:50.961 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-30T13:54:50.961 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110438 s, 949 MB/s 2024-03-30T13:54:50.961 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-30T13:54:51.539 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:51 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:54:51.539 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:51 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:54:51.539 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:51 smithi138 ceph-mon[33397]: osd.3 v2:172.21.15.102:6813/1080330048 boot 2024-03-30T13:54:51.539 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:51 smithi138 ceph-mon[33397]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T13:54:51.540 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:51 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:51.540 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:51 smithi138 ceph-mon[33397]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:51.571 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi138:vg_nvme/lv_4 2024-03-30T13:54:51.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:54:51.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:54:51.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[37717]: osd.3 v2:172.21.15.102:6813/1080330048 boot 2024-03-30T13:54:51.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[37717]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T13:54:51.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:51.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[37717]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:51.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:54:51.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:54:51.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[29392]: osd.3 v2:172.21.15.102:6813/1080330048 boot 2024-03-30T13:54:51.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[29392]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T13:54:51.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:54:51.635 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:51 smithi102 ceph-mon[29392]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-30T13:54:51.821 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi102 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi102 to 4016M 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:52.896 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:52.897 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:52 smithi138 ceph-mon[33397]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T13:54:53.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi102 2024-03-30T13:54:53.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi102 to 4016M 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[29392]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi102 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi102 to 4016M 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:54:53.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:52 smithi102 ceph-mon[37717]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T13:54:53.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/1303134689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd641011a0 msgr2=0x7fbd641015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/1303134689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd641011a0 0x7fbd641015c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbd600099c0 tx=0x7fbd60009cd0 comp rx=0 tx=0).stop 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/1303134689 shutdown_connections 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/1303134689 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd64102d40 0x7fbd641095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/1303134689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd641011a0 0x7fbd641015c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/1303134689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd641023a0 0x7fbd64102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/1303134689 >> 172.21.15.138:0/1303134689 conn(0x7fbd640fc760 msgr2=0x7fbd640feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/1303134689 shutdown_connections 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/1303134689 wait complete. 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 Processor -- start 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.353+0000 7fbd6ba2c700 1 -- start start 2024-03-30T13:54:53.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd641011a0 0x7fbd64105130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd641023a0 0x7fbd64103780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd64102d40 0x7fbd64103cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6410c4b0 con 0x7fbd641011a0 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6410c330 con 0x7fbd64102d40 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6410c630 con 0x7fbd641023a0 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd64102d40 0x7fbd64103cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd64102d40 0x7fbd64103cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:44998/0 (socket says 172.21.15.138:44998) 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 -- 172.21.15.138:0/2950764275 learned_addr learned my addr 172.21.15.138:0/2950764275 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd68fc7700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd641023a0 0x7fbd64103780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd697c8700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd641011a0 0x7fbd64105130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 -- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd641023a0 msgr2=0x7fbd64103780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd641023a0 0x7fbd64103780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:53.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 -- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd641011a0 msgr2=0x7fbd64105130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd641011a0 0x7fbd64105130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 -- 172.21.15.138:0/2950764275 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd64104570 con 0x7fbd64102d40 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd69fc9700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd64102d40 0x7fbd64103cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c007d70 tx=0x7fbd5c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd5a7fc700 1 -- 172.21.15.138:0/2950764275 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c016020 con 0x7fbd64102d40 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd641a1c50 con 0x7fbd64102d40 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd5a7fc700 1 -- 172.21.15.138:0/2950764275 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fbd5c00ac90 con 0x7fbd64102d40 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd5a7fc700 1 -- 172.21.15.138:0/2950764275 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c0115e0 con 0x7fbd64102d40 2024-03-30T13:54:53.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.354+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd641a2160 con 0x7fbd64102d40 2024-03-30T13:54:53.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.355+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd64068f00 con 0x7fbd64102d40 2024-03-30T13:54:53.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.356+0000 7fbd5a7fc700 1 -- 172.21.15.138:0/2950764275 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7fbd5c019020 con 0x7fbd64102d40 2024-03-30T13:54:53.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.356+0000 7fbd5a7fc700 1 --2- 172.21.15.138:0/2950764275 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbd5006db10 0x7fbd5006ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:54:53.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.356+0000 7fbd5a7fc700 1 -- 172.21.15.138:0/2950764275 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 2803+0+0 (secure 0 0 0) 0x7fbd5c08cbe0 con 0x7fbd64102d40 2024-03-30T13:54:53.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.356+0000 7fbd697c8700 1 --2- 172.21.15.138:0/2950764275 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbd5006db10 0x7fbd5006ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:54:53.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.357+0000 7fbd697c8700 1 --2- 172.21.15.138:0/2950764275 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbd5006db10 0x7fbd5006ffd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbd60000c00 tx=0x7fbd60005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:54:53.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.361+0000 7fbd5a7fc700 1 -- 172.21.15.138:0/2950764275 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbd5c020020 con 0x7fbd64102d40 2024-03-30T13:54:53.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:54:53.563+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fbd641a2390 con 0x7fbd5006db10 2024-03-30T13:54:53.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:53 smithi138 ceph-mon[33397]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:53.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:53.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:53.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:54.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[29392]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:54.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:54.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:54.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:54.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[37717]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:54.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:54:54.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:54:54.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:54:55.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:54 smithi138 ceph-mon[33397]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:55.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:54 smithi102 ceph-mon[29392]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:55.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:54 smithi102 ceph-mon[37717]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:54:56.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:55 smithi138 ceph-mon[33397]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:56.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:55 smithi102 ceph-mon[29392]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:56.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:55 smithi102 ceph-mon[37717]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:57.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/2602533768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]: dispatch 2024-03-30T13:54:57.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]: dispatch 2024-03-30T13:54:57.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]': finished 2024-03-30T13:54:57.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:56 smithi138 ceph-mon[33397]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T13:54:57.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:56 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:54:57.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/2042957242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:57.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/2602533768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]: dispatch 2024-03-30T13:54:57.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]: dispatch 2024-03-30T13:54:57.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]': finished 2024-03-30T13:54:57.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[29392]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T13:54:57.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:54:57.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/2042957242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:57.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/2602533768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]: dispatch 2024-03-30T13:54:57.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]: dispatch 2024-03-30T13:54:57.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a69453af-c657-49a7-b759-bebd7c7fefac"}]': finished 2024-03-30T13:54:57.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[37717]: osdmap e27: 5 total, 4 up, 5 in 2024-03-30T13:54:57.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:54:57.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/2042957242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:54:58.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:54:57 smithi138 ceph-mon[33397]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:58.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:54:57 smithi102 ceph-mon[29392]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:54:58.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:54:57 smithi102 ceph-mon[37717]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:00.394 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:00 smithi138 ceph-mon[33397]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:00.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:00 smithi102 ceph-mon[29392]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:00.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:00 smithi102 ceph-mon[37717]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:02.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:02 smithi102 ceph-mon[29392]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:02.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:02 smithi102 ceph-mon[37717]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:02.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:02 smithi138 ceph-mon[33397]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:03.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T13:55:03.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:03.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T13:55:03.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:03.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T13:55:03.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:04.354 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:04 smithi138 ceph-mon[33397]: Deploying daemon osd.4 on smithi138 2024-03-30T13:55:04.355 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:04 smithi138 ceph-mon[33397]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:04 smithi102 ceph-mon[29392]: Deploying daemon osd.4 on smithi138 2024-03-30T13:55:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:04 smithi102 ceph-mon[29392]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:04.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:04 smithi102 ceph-mon[37717]: Deploying daemon osd.4 on smithi138 2024-03-30T13:55:04.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:04 smithi102 ceph-mon[37717]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:06.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:06 smithi138 ceph-mon[33397]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:06.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:06 smithi102 ceph-mon[29392]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:06.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:06 smithi102 ceph-mon[37717]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:08.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:08 smithi138 ceph-mon[33397]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:08.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:08.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:08.338 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[29392]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[37717]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:08.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:10.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:10 smithi138 ceph-mon[33397]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:10.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:10 smithi102 ceph-mon[29392]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:10.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:10 smithi102 ceph-mon[37717]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:11.173 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:11 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.173 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:11 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.173 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:11 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:11.173 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:11 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:11.173 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:11 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.173 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:11 smithi138 ceph-mon[33397]: from='osd.4 v2:172.21.15.138:6800/1718735591' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:55:11.173 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:11 smithi138 ceph-mon[33397]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:55:11.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.174+0000 7fbd5a7fc700 1 -- 172.21.15.138:0/2950764275 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbd641a2390 con 0x7fbd5006db10 2024-03-30T13:55:11.174 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 4 on host 'smithi138' 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbd5006db10 msgr2=0x7fbd5006ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/2950764275 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbd5006db10 0x7fbd5006ffd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbd60000c00 tx=0x7fbd60005fb0 comp rx=0 tx=0).stop 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd64102d40 msgr2=0x7fbd64103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd64102d40 0x7fbd64103cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c007d70 tx=0x7fbd5c00a340 comp rx=0 tx=0).stop 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 shutdown_connections 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/2950764275 >> v2:172.21.15.102:6800/3154847229 conn(0x7fbd5006db10 0x7fbd5006ffd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd641011a0 0x7fbd64105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd64102d40 0x7fbd64103cc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 --2- 172.21.15.138:0/2950764275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd641023a0 0x7fbd64103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 >> 172.21.15.138:0/2950764275 conn(0x7fbd640fc760 msgr2=0x7fbd640fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:55:11.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 shutdown_connections 2024-03-30T13:55:11.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:11.177+0000 7fbd6ba2c700 1 -- 172.21.15.138:0/2950764275 wait complete. 2024-03-30T13:55:11.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:11.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:11.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[29392]: from='osd.4 v2:172.21.15.138:6800/1718735591' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[29392]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[37717]: from='osd.4 v2:172.21.15.138:6800/1718735591' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:55:11.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:11 smithi102 ceph-mon[37717]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T13:55:11.775 DEBUG:teuthology.orchestra.run.smithi138:osd.4> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T13:55:11.777 INFO:tasks.cephadm:Deploying osd.5 on smithi138 with /dev/vg_nvme/lv_3... 2024-03-30T13:55:11.777 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T13:55:12.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:12.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T13:55:12.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T13:55:12.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: from='osd.4 v2:172.21.15.138:6800/1718735591' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:12.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:12.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:12.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:12.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:12.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:12.437 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T13:55:12.125+0000 7fdb48ac7700 -1 osd.4 0 waiting for initial osdmap 2024-03-30T13:55:12.437 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 13:55:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T13:55:12.130+0000 7fdb448ba700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:55:12.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: from='osd.4 v2:172.21.15.138:6800/1718735591' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T13:55:12.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: osdmap e28: 5 total, 4 up, 5 in 2024-03-30T13:55:12.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: from='osd.4 v2:172.21.15.138:6800/1718735591' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:12.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:12.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:12.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:12.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:12.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:12 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:12.496 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:55:13.168 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:13 smithi138 ceph-mon[33397]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:55:13.168 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:13 smithi138 ceph-mon[33397]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T13:55:13.168 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:13 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:13.169 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:13 smithi138 ceph-mon[33397]: osd.4 v2:172.21.15.138:6800/1718735591 boot 2024-03-30T13:55:13.169 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:13 smithi138 ceph-mon[33397]: osdmap e30: 5 total, 5 up, 5 in 2024-03-30T13:55:13.169 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:13 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[29392]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[29392]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[29392]: osd.4 v2:172.21.15.138:6800/1718735591 boot 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[29392]: osdmap e30: 5 total, 5 up, 5 in 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[37717]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[37717]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T13:55:13.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[37717]: osd.4 v2:172.21.15.138:6800/1718735591 boot 2024-03-30T13:55:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[37717]: osdmap e30: 5 total, 5 up, 5 in 2024-03-30T13:55:13.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:13 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:55:14.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:55:14.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:55:14.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[29392]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:14.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:14 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:55:14.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:14 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:55:14.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:14 smithi138 ceph-mon[33397]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:14.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:14 smithi138 ceph-mon[33397]: osdmap e31: 5 total, 5 up, 5 in 2024-03-30T13:55:14.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[29392]: osdmap e31: 5 total, 5 up, 5 in 2024-03-30T13:55:14.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:55:14.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:55:14.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[37717]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-30T13:55:14.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:14 smithi102 ceph-mon[37717]: osdmap e31: 5 total, 5 up, 5 in 2024-03-30T13:55:14.861 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T13:55:14.862 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T13:55:14.862 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-30T13:55:14.862 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-30T13:55:14.862 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126039 s, 832 MB/s 2024-03-30T13:55:14.862 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-30T13:55:15.552 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi138:vg_nvme/lv_3 2024-03-30T13:55:16.169 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:15 smithi138 ceph-mon[33397]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T13:55:16.170 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:15 smithi138 ceph-mon[33397]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:16.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:15 smithi102 ceph-mon[37717]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T13:55:16.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:15 smithi102 ceph-mon[37717]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:16.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:15 smithi102 ceph-mon[29392]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T13:55:16.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:15 smithi102 ceph-mon[29392]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:16.319 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:17.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:17.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:17.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:17.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:17.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.810+0000 7fdd42922700 1 -- 172.21.15.138:0/193806306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 msgr2=0x7fdd3c107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:17.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.810+0000 7fdd42922700 1 --2- 172.21.15.138:0/193806306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c107a50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009990 tx=0x7fdd30009ca0 comp rx=0 tx=0).stop 2024-03-30T13:55:17.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 -- 172.21.15.138:0/193806306 shutdown_connections 2024-03-30T13:55:17.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 --2- 172.21.15.138:0/193806306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd3c102d30 0x7fdd3c105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:17.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 --2- 172.21.15.138:0/193806306 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd3c107f90 0x7fdd3c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 --2- 172.21.15.138:0/193806306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c107a50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 -- 172.21.15.138:0/193806306 >> 172.21.15.138:0/193806306 conn(0x7fdd3c0fc760 msgr2=0x7fdd3c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 -- 172.21.15.138:0/193806306 shutdown_connections 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 -- 172.21.15.138:0/193806306 wait complete. 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 Processor -- start 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 -- start start 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd3c102d30 0x7fdd3c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd3c107f90 0x7fdd3c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.811+0000 7fdd42922700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3c10d580 con 0x7fdd3c102d30 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd42922700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3c10d400 con 0x7fdd3c105660 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd42922700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3c10d700 con 0x7fdd3c107f90 2024-03-30T13:55:17.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34466/0 (socket says 172.21.15.138:34466) 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 -- 172.21.15.138:0/3195694696 learned_addr learned my addr 172.21.15.138:0/3195694696 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd40ebf700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd3c107f90 0x7fdd3c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3bfff700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd3c102d30 0x7fdd3c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 -- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd3c107f90 msgr2=0x7fdd3c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd3c107f90 0x7fdd3c19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 -- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd3c102d30 msgr2=0x7fdd3c197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd3c102d30 0x7fdd3c197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 -- 172.21.15.138:0/3195694696 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3c19f7e0 con 0x7fdd3c105660 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd3b7fe700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c1980c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b5c0 tx=0x7fdd300036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd397fa700 1 -- 172.21.15.138:0/3195694696 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30026070 con 0x7fdd3c105660 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3c19fa70 con 0x7fdd3c105660 2024-03-30T13:55:17.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd397fa700 1 -- 172.21.15.138:0/3195694696 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fdd30004ca0 con 0x7fdd3c105660 2024-03-30T13:55:17.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd397fa700 1 -- 172.21.15.138:0/3195694696 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30003df0 con 0x7fdd3c105660 2024-03-30T13:55:17.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.812+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd3c19ff20 con 0x7fdd3c105660 2024-03-30T13:55:17.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.813+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd3c068f00 con 0x7fdd3c105660 2024-03-30T13:55:17.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.814+0000 7fdd397fa700 1 -- 172.21.15.138:0/3195694696 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7fdd30029070 con 0x7fdd3c105660 2024-03-30T13:55:17.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.814+0000 7fdd397fa700 1 --2- 172.21.15.138:0/3195694696 >> v2:172.21.15.102:6800/3154847229 conn(0x7fdd2406db10 0x7fdd2406ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:17.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.814+0000 7fdd397fa700 1 -- 172.21.15.138:0/3195694696 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3205+0+0 (secure 0 0 0) 0x7fdd3008d790 con 0x7fdd3c105660 2024-03-30T13:55:17.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.814+0000 7fdd3bfff700 1 --2- 172.21.15.138:0/3195694696 >> v2:172.21.15.102:6800/3154847229 conn(0x7fdd2406db10 0x7fdd2406ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:17.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.815+0000 7fdd3bfff700 1 --2- 172.21.15.138:0/3195694696 >> v2:172.21.15.102:6800/3154847229 conn(0x7fdd2406db10 0x7fdd2406ffd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c005950 tx=0x7fdd2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:55:17.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:17.819+0000 7fdd397fa700 1 -- 172.21.15.138:0/3195694696 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdd30013070 con 0x7fdd3c105660 2024-03-30T13:55:18.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:18.026+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fdd3c062680 con 0x7fdd2406db10 2024-03-30T13:55:18.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:17 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi138 2024-03-30T13:55:18.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:17 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi138 to 17088M 2024-03-30T13:55:18.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:17 smithi138 ceph-mon[33397]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T13:55:18.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:17 smithi138 ceph-mon[33397]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:18.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi138 2024-03-30T13:55:18.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi138 to 17088M 2024-03-30T13:55:18.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[29392]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T13:55:18.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[29392]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:18.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi138 2024-03-30T13:55:18.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi138 to 17088M 2024-03-30T13:55:18.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[37717]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T13:55:18.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:17 smithi102 ceph-mon[37717]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:19.177 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:18 smithi138 ceph-mon[33397]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:55:19.177 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:55:19.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:55:19.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:19.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[29392]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:55:19.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:55:19.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:55:19.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[37717]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:55:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:55:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:55:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:20.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:19 smithi138 ceph-mon[33397]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-30T13:55:20.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:19 smithi102 ceph-mon[29392]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-30T13:55:20.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:19 smithi102 ceph-mon[37717]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-30T13:55:21.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/4046919717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]: dispatch 2024-03-30T13:55:21.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]: dispatch 2024-03-30T13:55:21.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]': finished 2024-03-30T13:55:21.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:20 smithi138 ceph-mon[33397]: osdmap e35: 6 total, 5 up, 6 in 2024-03-30T13:55:21.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:20 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/4046919717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]: dispatch 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]: dispatch 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]': finished 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[29392]: osdmap e35: 6 total, 5 up, 6 in 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/4046919717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]: dispatch 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]: dispatch 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b723775-3bc5-4c26-8c7b-570cfe9a270d"}]': finished 2024-03-30T13:55:21.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[37717]: osdmap e35: 6 total, 5 up, 6 in 2024-03-30T13:55:21.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:20 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:22.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:21 smithi138 ceph-mon[33397]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:55:22.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/2914651816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:55:22.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:21 smithi102 ceph-mon[37717]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:55:22.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/2914651816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:55:22.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:21 smithi102 ceph-mon[29392]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:55:22.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/2914651816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:55:24.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:24 smithi138 ceph-mon[33397]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T13:55:24.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:24 smithi102 ceph-mon[29392]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T13:55:24.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:24 smithi102 ceph-mon[37717]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T13:55:26.417 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:26 smithi138 ceph-mon[33397]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T13:55:26.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:26 smithi102 ceph-mon[29392]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T13:55:26.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:26 smithi102 ceph-mon[37717]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T13:55:27.368 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T13:55:27.369 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T13:55:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:27.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T13:55:27.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:28.371 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:28 smithi138 ceph-mon[33397]: Deploying daemon osd.5 on smithi138 2024-03-30T13:55:28.371 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:28 smithi138 ceph-mon[33397]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:55:28.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:28 smithi102 ceph-mon[29392]: Deploying daemon osd.5 on smithi138 2024-03-30T13:55:28.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:28 smithi102 ceph-mon[29392]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:55:28.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:28 smithi102 ceph-mon[37717]: Deploying daemon osd.5 on smithi138 2024-03-30T13:55:28.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:28 smithi102 ceph-mon[37717]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:55:30.340 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:30 smithi138 ceph-mon[33397]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:30.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:30 smithi102 ceph-mon[29392]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:30.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:30 smithi102 ceph-mon[37717]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:32.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:32 smithi138 ceph-mon[33397]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:32.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:32 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:32.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:32 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:32.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:32 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[29392]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[37717]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:32 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:34.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:34 smithi138 ceph-mon[33397]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:34.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:34 smithi102 ceph-mon[29392]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:34.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:34 smithi102 ceph-mon[37717]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: from='osd.5 v2:172.21.15.138:6804/1568298628' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:55:35.586 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:35 smithi138 ceph-mon[33397]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:55:35.635 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 5 on host 'smithi138' 2024-03-30T13:55:35.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.638+0000 7fdd397fa700 1 -- 172.21.15.138:0/3195694696 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdd3c062680 con 0x7fdd2406db10 2024-03-30T13:55:35.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 >> v2:172.21.15.102:6800/3154847229 conn(0x7fdd2406db10 msgr2=0x7fdd2406ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:35.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 --2- 172.21.15.138:0/3195694696 >> v2:172.21.15.102:6800/3154847229 conn(0x7fdd2406db10 0x7fdd2406ffd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c005950 tx=0x7fdd2c00b410 comp rx=0 tx=0).stop 2024-03-30T13:55:35.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 msgr2=0x7fdd3c1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c1980c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b5c0 tx=0x7fdd300036f0 comp rx=0 tx=0).stop 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 shutdown_connections 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 --2- 172.21.15.138:0/3195694696 >> v2:172.21.15.102:6800/3154847229 conn(0x7fdd2406db10 0x7fdd2406ffd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd3c102d30 0x7fdd3c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd3c107f90 0x7fdd3c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 --2- 172.21.15.138:0/3195694696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd3c105660 0x7fdd3c1980c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 >> 172.21.15.138:0/3195694696 conn(0x7fdd3c0fc760 msgr2=0x7fdd3c1039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 shutdown_connections 2024-03-30T13:55:35.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:35.640+0000 7fdd42922700 1 -- 172.21.15.138:0/3195694696 wait complete. 2024-03-30T13:55:35.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: from='osd.5 v2:172.21.15.138:6804/1568298628' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[29392]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:35.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:35.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:35.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:35.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: from='osd.5 v2:172.21.15.138:6804/1568298628' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:55:35.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:35 smithi102 ceph-mon[37717]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T13:55:36.240 DEBUG:teuthology.orchestra.run.smithi138:osd.5> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-30T13:55:36.242 INFO:tasks.cephadm:Deploying osd.6 on smithi138 with /dev/vg_nvme/lv_2... 2024-03-30T13:55:36.242 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T13:55:36.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T13:55:36.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T13:55:36.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: from='osd.5 v2:172.21.15.138:6804/1568298628' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:36.743 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:36.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:36.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:36.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:36.744 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:36.744 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T13:55:36.555+0000 7fc804788700 -1 osd.5 0 waiting for initial osdmap 2024-03-30T13:55:36.744 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 13:55:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T13:55:36.561+0000 7fc80057b700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: from='osd.5 v2:172.21.15.138:6804/1568298628' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T13:55:36.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T13:55:36.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: from='osd.5 v2:172.21.15.138:6804/1568298628' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:36.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:36.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:55:36.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:36.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:36.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:36 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:36.967 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:55:37.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:37 smithi138 ceph-mon[33397]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:55:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:37 smithi138 ceph-mon[33397]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T13:55:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:37 smithi138 ceph-mon[33397]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:37 smithi138 ceph-mon[33397]: osd.5 v2:172.21.15.138:6804/1568298628 boot 2024-03-30T13:55:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:37 smithi138 ceph-mon[33397]: osdmap e38: 6 total, 6 up, 6 in 2024-03-30T13:55:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:37 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[29392]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[29392]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[29392]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[29392]: osd.5 v2:172.21.15.138:6804/1568298628 boot 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[29392]: osdmap e38: 6 total, 6 up, 6 in 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[37717]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:55:37.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[37717]: osdmap e37: 6 total, 5 up, 6 in 2024-03-30T13:55:37.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:37.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[37717]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-30T13:55:37.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[37717]: osd.5 v2:172.21.15.138:6804/1568298628 boot 2024-03-30T13:55:37.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[37717]: osdmap e38: 6 total, 6 up, 6 in 2024-03-30T13:55:37.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:37 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:55:38.857 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:38 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:55:38.857 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:38 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:55:38.857 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:38 smithi138 ceph-mon[33397]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T13:55:38.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:38 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:55:38.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:38 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:55:38.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:38 smithi102 ceph-mon[29392]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T13:55:38.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:38 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:55:38.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:38 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:55:38.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:38 smithi102 ceph-mon[37717]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T13:55:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T13:55:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T13:55:39.558 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-30T13:55:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-30T13:55:39.558 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135484 s, 774 MB/s 2024-03-30T13:55:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-30T13:55:39.890 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:39 smithi138 ceph-mon[33397]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:39.890 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:39 smithi138 ceph-mon[33397]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T13:55:39.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:39 smithi102 ceph-mon[29392]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:39.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:39 smithi102 ceph-mon[29392]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T13:55:39.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:39 smithi102 ceph-mon[37717]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:39.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:39 smithi102 ceph-mon[37717]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T13:55:40.210 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi138:vg_nvme/lv_2 2024-03-30T13:55:40.879 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:55:41.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi138 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi138 to 8544M 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:41 smithi138 ceph-mon[33397]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi138 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi138 to 8544M 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[29392]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi138 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi138 to 8544M 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:41.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:41 smithi102 ceph-mon[37717]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:42.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.478+0000 7f2ea79a6700 1 -- 172.21.15.138:0/3768339187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01011e0 msgr2=0x7f2ea0101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:42.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.478+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/3768339187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01011e0 0x7f2ea0101600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0099c0 tx=0x7f2e9c009cd0 comp rx=0 tx=0).stop 2024-03-30T13:55:42.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.478+0000 7f2ea79a6700 1 -- 172.21.15.138:0/3768339187 shutdown_connections 2024-03-30T13:55:42.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/3768339187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea0102d80 0x7f2ea0109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:42.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/3768339187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01011e0 0x7f2ea0101600 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:42.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/3768339187 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2ea01023e0 0x7f2ea0102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 -- 172.21.15.138:0/3768339187 >> 172.21.15.138:0/3768339187 conn(0x7f2ea00fc7a0 msgr2=0x7f2ea00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 -- 172.21.15.138:0/3768339187 shutdown_connections 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 -- 172.21.15.138:0/3768339187 wait complete. 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 Processor -- start 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 -- start start 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea01011e0 0x7f2ea0196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01023e0 0x7f2ea0196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2ea0102d80 0x7f2ea019dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea010c480 con 0x7f2ea01011e0 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea010c300 con 0x7f2ea01023e0 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.479+0000 7f2ea79a6700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea010c600 con 0x7f2ea0102d80 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01023e0 0x7f2ea0196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01023e0 0x7f2ea0196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46600/0 (socket says 172.21.15.138:46600) 2024-03-30T13:55:42.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 -- 172.21.15.138:0/1788706526 learned_addr learned my addr 172.21.15.138:0/1788706526 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea5742700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea01011e0 0x7f2ea0196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea5f43700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2ea0102d80 0x7f2ea019dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 -- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2ea0102d80 msgr2=0x7f2ea019dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2ea0102d80 0x7f2ea019dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 -- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea01011e0 msgr2=0x7f2ea0196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea01011e0 0x7f2ea0196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 -- 172.21.15.138:0/1788706526 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea019e690 con 0x7f2ea01023e0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea5f43700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2ea0102d80 0x7f2ea019dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea4f41700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01023e0 0x7f2ea0196f70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000c9b0 tx=0x7f2e9000ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2e967fc700 1 -- 172.21.15.138:0/1788706526 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90007b00 con 0x7f2ea01023e0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea019e980 con 0x7f2ea01023e0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2e967fc700 1 -- 172.21.15.138:0/1788706526 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f2e90007ca0 con 0x7f2ea01023e0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2e967fc700 1 -- 172.21.15.138:0/1788706526 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90018660 con 0x7f2ea01023e0 2024-03-30T13:55:42.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.480+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea019ef10 con 0x7f2ea01023e0 2024-03-30T13:55:42.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.481+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea0068f00 con 0x7f2ea01023e0 2024-03-30T13:55:42.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.482+0000 7f2e967fc700 1 -- 172.21.15.138:0/1788706526 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f2e9000f450 con 0x7f2ea01023e0 2024-03-30T13:55:42.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.482+0000 7f2e967fc700 1 --2- 172.21.15.138:0/1788706526 >> v2:172.21.15.102:6800/3154847229 conn(0x7f2e8c06db10 0x7f2e8c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:55:42.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.482+0000 7f2e967fc700 1 -- 172.21.15.138:0/1788706526 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 3497+0+0 (secure 0 0 0) 0x7f2e9008cf80 con 0x7f2ea01023e0 2024-03-30T13:55:42.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.483+0000 7f2ea5742700 1 --2- 172.21.15.138:0/1788706526 >> v2:172.21.15.102:6800/3154847229 conn(0x7f2e8c06db10 0x7f2e8c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:55:42.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.483+0000 7f2ea5742700 1 --2- 172.21.15.138:0/1788706526 >> v2:172.21.15.102:6800/3154847229 conn(0x7f2e8c06db10 0x7f2e8c06ffd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009990 tx=0x7f2e9c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:55:42.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.487+0000 7f2e967fc700 1 -- 172.21.15.138:0/1788706526 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2e90014030 con 0x7f2ea01023e0 2024-03-30T13:55:42.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:55:42.693+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2ea002de60 con 0x7f2e8c06db10 2024-03-30T13:55:42.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:42 smithi138 ceph-mon[33397]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T13:55:42.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:42 smithi102 ceph-mon[29392]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T13:55:42.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:42 smithi102 ceph-mon[37717]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T13:55:43.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:43 smithi138 ceph-mon[33397]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:55:43.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:43 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:55:43.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:43 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:55:43.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:43 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:43.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:43 smithi138 ceph-mon[33397]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-30T13:55:43.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[29392]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:55:43.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:55:43.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:55:43.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:43.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[29392]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-30T13:55:43.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[37717]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:55:43.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:55:43.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:55:43.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:43.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:43 smithi102 ceph-mon[37717]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-30T13:55:46.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:46 smithi138 ceph-mon[33397]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:55:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/1526295695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]: dispatch 2024-03-30T13:55:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]: dispatch 2024-03-30T13:55:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]': finished 2024-03-30T13:55:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:46 smithi138 ceph-mon[33397]: osdmap e43: 7 total, 6 up, 7 in 2024-03-30T13:55:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:46 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:55:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/4018330625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:55:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[29392]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:55:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/1526295695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]: dispatch 2024-03-30T13:55:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]: dispatch 2024-03-30T13:55:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]': finished 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[29392]: osdmap e43: 7 total, 6 up, 7 in 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/4018330625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[37717]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/1526295695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]: dispatch 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]: dispatch 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cdfc0c0-2f91-44d9-9c0a-401e328da8af"}]': finished 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[37717]: osdmap e43: 7 total, 6 up, 7 in 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:55:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/4018330625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:55:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:48 smithi138 ceph-mon[33397]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-30T13:55:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:48 smithi102 ceph-mon[29392]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-30T13:55:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:48 smithi102 ceph-mon[37717]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-30T13:55:50.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:50 smithi138 ceph-mon[33397]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:55:50.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:50 smithi102 ceph-mon[29392]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:55:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:50 smithi102 ceph-mon[37717]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T13:55:52.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:52 smithi138 ceph-mon[33397]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-30T13:55:52.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T13:55:52.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:52 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:52.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:52 smithi102 ceph-mon[29392]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-30T13:55:52.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T13:55:52.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:52 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:52 smithi102 ceph-mon[37717]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-30T13:55:52.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T13:55:52.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:52 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:55:53.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:53 smithi138 ceph-mon[33397]: Deploying daemon osd.6 on smithi138 2024-03-30T13:55:53.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:53 smithi102 ceph-mon[29392]: Deploying daemon osd.6 on smithi138 2024-03-30T13:55:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:53 smithi102 ceph-mon[37717]: Deploying daemon osd.6 on smithi138 2024-03-30T13:55:54.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:54 smithi138 ceph-mon[33397]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:54.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:54 smithi102 ceph-mon[29392]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:54.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:54 smithi102 ceph-mon[37717]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:56.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:56 smithi138 ceph-mon[33397]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:56.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:56 smithi102 ceph-mon[29392]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:56.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:56 smithi102 ceph-mon[37717]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:58.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:58 smithi138 ceph-mon[33397]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:58.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:58.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:58.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:55:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[29392]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:58.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[37717]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:55:58.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:55:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:55:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:55:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:00.220 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:00 smithi138 ceph-mon[33397]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:00.220 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:00.220 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:00 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:00.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:00 smithi102 ceph-mon[29392]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:00.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:00.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:00 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:00.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:00 smithi102 ceph-mon[37717]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:00.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:00.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:00 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:00.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.655+0000 7f2e967fc700 1 -- 172.21.15.138:0/1788706526 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2ea002de60 con 0x7f2e8c06db10 2024-03-30T13:56:00.653 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 6 on host 'smithi138' 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 >> v2:172.21.15.102:6800/3154847229 conn(0x7f2e8c06db10 msgr2=0x7f2e8c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/1788706526 >> v2:172.21.15.102:6800/3154847229 conn(0x7f2e8c06db10 0x7f2e8c06ffd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009990 tx=0x7f2e9c005fb0 comp rx=0 tx=0).stop 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01023e0 msgr2=0x7f2ea0196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01023e0 0x7f2ea0196f70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000c9b0 tx=0x7f2e9000ccc0 comp rx=0 tx=0).stop 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 shutdown_connections 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/1788706526 >> v2:172.21.15.102:6800/3154847229 conn(0x7f2e8c06db10 0x7f2e8c06ffd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea01011e0 0x7f2ea0196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2ea0102d80 0x7f2ea019dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:00.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 --2- 172.21.15.138:0/1788706526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ea01023e0 0x7f2ea0196f70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:00.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 >> 172.21.15.138:0/1788706526 conn(0x7f2ea00fc7a0 msgr2=0x7f2ea0105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:00.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 shutdown_connections 2024-03-30T13:56:00.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:00.658+0000 7f2ea79a6700 1 -- 172.21.15.138:0/1788706526 wait complete. 2024-03-30T13:56:01.235 DEBUG:teuthology.orchestra.run.smithi138:osd.6> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.6.service 2024-03-30T13:56:01.237 INFO:tasks.cephadm:Deploying osd.7 on smithi138 with /dev/vg_nvme/lv_1... 2024-03-30T13:56:01.237 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T13:56:01.259 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='osd.6 v2:172.21.15.138:6808/3214961690' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:56:01.260 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:56:01.260 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:01.260 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:01.260 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.260 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.260 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.260 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:01 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:01.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='osd.6 v2:172.21.15.138:6808/3214961690' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:56:01.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:56:01.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='osd.6 v2:172.21.15.138:6808/3214961690' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:01.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:01 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:02.208 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: from='osd.6 v2:172.21.15.138:6808/3214961690' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: osdmap e44: 7 total, 6 up, 7 in 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:56:02.330 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: osdmap e45: 7 total, 6 up, 7 in 2024-03-30T13:56:02.331 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.331 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.331 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T13:56:02.112+0000 7f3e12c2c700 -1 osd.6 0 waiting for initial osdmap 2024-03-30T13:56:02.331 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 13:56:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T13:56:02.118+0000 7f3e0ea1f700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:56:02.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: from='osd.6 v2:172.21.15.138:6808/3214961690' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: osdmap e44: 7 total, 6 up, 7 in 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: osdmap e45: 7 total, 6 up, 7 in 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: from='osd.6 v2:172.21.15.138:6808/3214961690' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: osdmap e44: 7 total, 6 up, 7 in 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: osdmap e45: 7 total, 6 up, 7 in 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:02.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:02 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:04.266 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:04 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:56:04.266 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:04 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:56:04.266 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:04 smithi138 ceph-mon[33397]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:04.266 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:04 smithi138 ceph-mon[33397]: osd.6 v2:172.21.15.138:6808/3214961690 boot 2024-03-30T13:56:04.266 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:04 smithi138 ceph-mon[33397]: osdmap e46: 7 total, 7 up, 7 in 2024-03-30T13:56:04.266 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:04 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:04.266 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:04 smithi138 ceph-mon[33397]: osdmap e47: 7 total, 7 up, 7 in 2024-03-30T13:56:04.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:56:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:56:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[29392]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[29392]: osd.6 v2:172.21.15.138:6808/3214961690 boot 2024-03-30T13:56:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[29392]: osdmap e46: 7 total, 7 up, 7 in 2024-03-30T13:56:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:04.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[29392]: osdmap e47: 7 total, 7 up, 7 in 2024-03-30T13:56:04.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:56:04.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:56:04.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[37717]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-30T13:56:04.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[37717]: osd.6 v2:172.21.15.138:6808/3214961690 boot 2024-03-30T13:56:04.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[37717]: osdmap e46: 7 total, 7 up, 7 in 2024-03-30T13:56:04.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:56:04.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:04 smithi102 ceph-mon[37717]: osdmap e47: 7 total, 7 up, 7 in 2024-03-30T13:56:04.707 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T13:56:04.707 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T13:56:04.707 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-03-30T13:56:04.707 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-03-30T13:56:04.707 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123037 s, 852 MB/s 2024-03-30T13:56:04.707 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-03-30T13:56:05.366 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi138:vg_nvme/lv_1 2024-03-30T13:56:05.601 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi138 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi138 to 5696M 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:05.984 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:05.985 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:05.985 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:05.985 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: osdmap e48: 7 total, 7 up, 7 in 2024-03-30T13:56:05.985 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:05 smithi138 ceph-mon[33397]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi138 2024-03-30T13:56:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi138 to 5696M 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: osdmap e48: 7 total, 7 up, 7 in 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[29392]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi138 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi138 to 5696M 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: osdmap e48: 7 total, 7 up, 7 in 2024-03-30T13:56:06.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:05 smithi102 ceph-mon[37717]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 -- 172.21.15.138:0/3783238290 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103e30 msgr2=0x7fc84810a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 --2- 172.21.15.138:0/3783238290 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103e30 0x7fc84810a6e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc8400099c0 tx=0x7fc840009cd0 comp rx=0 tx=0).stop 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 -- 172.21.15.138:0/3783238290 shutdown_connections 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 --2- 172.21.15.138:0/3783238290 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103e30 0x7fc84810a6e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 --2- 172.21.15.138:0/3783238290 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc848102290 0x7fc8481026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 --2- 172.21.15.138:0/3783238290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc848103490 0x7fc8481038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 -- 172.21.15.138:0/3783238290 >> 172.21.15.138:0/3783238290 conn(0x7fc8480fd850 msgr2=0x7fc8480ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:07.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 -- 172.21.15.138:0/3783238290 shutdown_connections 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.072+0000 7fc84d97c700 1 -- 172.21.15.138:0/3783238290 wait complete. 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 Processor -- start 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 -- start start 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc848102290 0x7fc848197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103490 0x7fc848198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc848103e30 0x7fc84819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810d600 con 0x7fc848102290 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc84810d480 con 0x7fc848103490 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc84d97c700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc84810d780 con 0x7fc848103e30 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103490 0x7fc848198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103490 0x7fc848198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40238/0 (socket says 172.21.15.138:40238) 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 -- 172.21.15.138:0/1550390256 learned_addr learned my addr 172.21.15.138:0/1550390256 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc846ffd700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc848102290 0x7fc848197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:07.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8477fe700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc848103e30 0x7fc84819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 -- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc848103e30 msgr2=0x7fc84819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc848103e30 0x7fc84819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 -- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc848102290 msgr2=0x7fc848197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc848102290 0x7fc848197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8467fc700 1 -- 172.21.15.138:0/1550390256 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84819f790 con 0x7fc848103490 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.073+0000 7fc8477fe700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc848103e30 0x7fc84819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.074+0000 7fc8467fc700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103490 0x7fc848198070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc83000d950 tx=0x7fc83000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.074+0000 7fc84c97a700 1 -- 172.21.15.138:0/1550390256 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830016070 con 0x7fc848103490 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.074+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84819fa80 con 0x7fc848103490 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.074+0000 7fc84c97a700 1 -- 172.21.15.138:0/1550390256 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fc830009df0 con 0x7fc848103490 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.074+0000 7fc84c97a700 1 -- 172.21.15.138:0/1550390256 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83000f3c0 con 0x7fc848103490 2024-03-30T13:56:07.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.074+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8481a0010 con 0x7fc848103490 2024-03-30T13:56:07.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.075+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc848068f00 con 0x7fc848103490 2024-03-30T13:56:07.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.075+0000 7fc84c97a700 1 -- 172.21.15.138:0/1550390256 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7fc830019070 con 0x7fc848103490 2024-03-30T13:56:07.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.076+0000 7fc84c97a700 1 --2- 172.21.15.138:0/1550390256 >> v2:172.21.15.102:6800/3154847229 conn(0x7fc83406db10 0x7fc83406ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:07.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.076+0000 7fc84c97a700 1 -- 172.21.15.138:0/1550390256 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 3789+0+0 (secure 0 0 0) 0x7fc83008cbd0 con 0x7fc848103490 2024-03-30T13:56:07.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.076+0000 7fc846ffd700 1 --2- 172.21.15.138:0/1550390256 >> v2:172.21.15.102:6800/3154847229 conn(0x7fc83406db10 0x7fc83406ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:07.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.076+0000 7fc846ffd700 1 --2- 172.21.15.138:0/1550390256 >> v2:172.21.15.102:6800/3154847229 conn(0x7fc83406db10 0x7fc83406ffd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8481032f0 tx=0x7fc838009180 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:07.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.080+0000 7fc84c97a700 1 -- 172.21.15.138:0/1550390256 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc83005b060 con 0x7fc848103490 2024-03-30T13:56:07.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:06 smithi138 ceph-mon[33397]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T13:56:07.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:06 smithi102 ceph-mon[29392]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T13:56:07.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:06 smithi102 ceph-mon[37717]: osdmap e49: 7 total, 7 up, 7 in 2024-03-30T13:56:07.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:07.285+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc848062680 con 0x7fc83406db10 2024-03-30T13:56:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:07 smithi138 ceph-mon[33397]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T13:56:08.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:07 smithi138 ceph-mon[33397]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:08.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:07 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:56:08.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:07 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:56:08.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:07 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:08.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[29392]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T13:56:08.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[29392]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:08.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:56:08.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:56:08.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:08.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[37717]: osdmap e50: 7 total, 7 up, 7 in 2024-03-30T13:56:08.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[37717]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:08.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T13:56:08.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T13:56:08.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:07 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:09.138 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:08 smithi138 ceph-mon[33397]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:09.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:08 smithi102 ceph-mon[29392]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:09.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:08 smithi102 ceph-mon[37717]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:10.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:09 smithi138 ceph-mon[33397]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-30T13:56:10.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:09 smithi102 ceph-mon[29392]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-30T13:56:10.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:09 smithi102 ceph-mon[37717]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-30T13:56:11.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/3431780124' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]: dispatch 2024-03-30T13:56:11.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]: dispatch 2024-03-30T13:56:11.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]': finished 2024-03-30T13:56:11.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:10 smithi138 ceph-mon[33397]: osdmap e51: 8 total, 7 up, 8 in 2024-03-30T13:56:11.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:10 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:11.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/3931411288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:56:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/3431780124' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]: dispatch 2024-03-30T13:56:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]: dispatch 2024-03-30T13:56:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]': finished 2024-03-30T13:56:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[37717]: osdmap e51: 8 total, 7 up, 8 in 2024-03-30T13:56:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/3931411288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:56:11.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/3431780124' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]: dispatch 2024-03-30T13:56:11.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]: dispatch 2024-03-30T13:56:11.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "67ed0818-2486-420e-97ff-0264fac6a48b"}]': finished 2024-03-30T13:56:11.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[29392]: osdmap e51: 8 total, 7 up, 8 in 2024-03-30T13:56:11.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:11.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/3931411288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T13:56:12.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:11 smithi138 ceph-mon[33397]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:56:12.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:11 smithi102 ceph-mon[29392]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:56:12.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:11 smithi102 ceph-mon[37717]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:56:14.344 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:14 smithi138 ceph-mon[33397]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T13:56:14.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:14 smithi102 ceph-mon[37717]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T13:56:14.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:14 smithi102 ceph-mon[29392]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T13:56:16.331 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:16 smithi138 ceph-mon[33397]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T13:56:16.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:16 smithi102 ceph-mon[37717]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T13:56:16.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:16 smithi102 ceph-mon[29392]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T13:56:17.284 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:17 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T13:56:17.284 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:17 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:17.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:17 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T13:56:17.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:17 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:17.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:17 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T13:56:17.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:17 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:18.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:18 smithi138 ceph-mon[33397]: Deploying daemon osd.7 on smithi138 2024-03-30T13:56:18.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:18 smithi138 ceph-mon[33397]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:56:18.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:18 smithi102 ceph-mon[29392]: Deploying daemon osd.7 on smithi138 2024-03-30T13:56:18.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:18 smithi102 ceph-mon[29392]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:56:18.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:18 smithi102 ceph-mon[37717]: Deploying daemon osd.7 on smithi138 2024-03-30T13:56:18.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:18 smithi102 ceph-mon[37717]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:56:20.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:20 smithi138 ceph-mon[33397]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:20.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:20 smithi102 ceph-mon[29392]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:20.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:20 smithi102 ceph-mon[37717]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:22.342 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:22 smithi138 ceph-mon[33397]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:22.342 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:22 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:22.342 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:22 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:22.342 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:22 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:22.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[37717]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:22.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:22.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:22.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:22.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[29392]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:22.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:22.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:22.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:22 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:24.373 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:24 smithi138 ceph-mon[33397]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:24.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:24 smithi102 ceph-mon[29392]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:24 smithi102 ceph-mon[37717]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:24.745 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 7 on host 'smithi138' 2024-03-30T13:56:24.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.748+0000 7fc84c97a700 1 -- 172.21.15.138:0/1550390256 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc848062680 con 0x7fc83406db10 2024-03-30T13:56:24.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 >> v2:172.21.15.102:6800/3154847229 conn(0x7fc83406db10 msgr2=0x7fc83406ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:24.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 --2- 172.21.15.138:0/1550390256 >> v2:172.21.15.102:6800/3154847229 conn(0x7fc83406db10 0x7fc83406ffd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8481032f0 tx=0x7fc838009180 comp rx=0 tx=0).stop 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103490 msgr2=0x7fc848198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103490 0x7fc848198070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc83000d950 tx=0x7fc83000dc60 comp rx=0 tx=0).stop 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 shutdown_connections 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 --2- 172.21.15.138:0/1550390256 >> v2:172.21.15.102:6800/3154847229 conn(0x7fc83406db10 0x7fc83406ffd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc848102290 0x7fc848197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc848103e30 0x7fc84819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 --2- 172.21.15.138:0/1550390256 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc848103490 0x7fc848198070 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 >> 172.21.15.138:0/1550390256 conn(0x7fc8480fd850 msgr2=0x7fc848106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 shutdown_connections 2024-03-30T13:56:24.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:24.750+0000 7fc84d97c700 1 -- 172.21.15.138:0/1550390256 wait complete. 2024-03-30T13:56:25.526 DEBUG:teuthology.orchestra.run.smithi138:osd.7> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.7.service 2024-03-30T13:56:25.529 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-30T13:56:25.529 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T13:56:25.781 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[29392]: from='osd.7 v2:172.21.15.138:6812/3765635388' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[29392]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[29392]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[37717]: from='osd.7 v2:172.21.15.138:6812/3765635388' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:56:25.905 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[37717]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:56:25.906 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:25 smithi102 ceph-mon[37717]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:25.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:25 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:25.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:25 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:25.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:25 smithi138 ceph-mon[33397]: from='osd.7 v2:172.21.15.138:6812/3765635388' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:56:25.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:25 smithi138 ceph-mon[33397]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T13:56:25.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:25 smithi138 ceph-mon[33397]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:26.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-mon[33397]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T13:56:26.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-mon[33397]: from='osd.7 v2:172.21.15.138:6812/3765635388' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:26.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-mon[33397]: osdmap e52: 8 total, 7 up, 8 in 2024-03-30T13:56:26.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:26.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-mon[33397]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:26.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:26.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:26.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T13:56:26.732+0000 7f789ba03700 -1 osd.7 0 waiting for initial osdmap 2024-03-30T13:56:26.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 13:56:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T13:56:26.737+0000 7f78977f6700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:56:26.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[37717]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T13:56:26.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[37717]: from='osd.7 v2:172.21.15.138:6812/3765635388' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:26.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[37717]: osdmap e52: 8 total, 7 up, 8 in 2024-03-30T13:56:26.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:26.980 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[37717]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[29392]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[29392]: from='osd.7 v2:172.21.15.138:6812/3765635388' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[29392]: osdmap e52: 8 total, 7 up, 8 in 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[29392]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:26.981 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:26 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:27.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 -- 172.21.15.102:0/1913132019 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c101170 msgr2=0x7f0f1c101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:27.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 --2- 172.21.15.102:0/1913132019 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c101170 0x7f0f1c101590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c0098f0 tx=0x7f0f0c009c00 comp rx=0 tx=0).stop 2024-03-30T13:56:27.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 -- 172.21.15.102:0/1913132019 shutdown_connections 2024-03-30T13:56:27.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 --2- 172.21.15.102:0/1913132019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f1c102d10 0x7f0f1c1095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 --2- 172.21.15.102:0/1913132019 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c101170 0x7f0f1c101590 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 --2- 172.21.15.102:0/1913132019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 0x7f0f1c1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 -- 172.21.15.102:0/1913132019 >> 172.21.15.102:0/1913132019 conn(0x7f0f1c0fc730 msgr2=0x7f0f1c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 -- 172.21.15.102:0/1913132019 shutdown_connections 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.489+0000 7f0f2410f700 1 -- 172.21.15.102:0/1913132019 wait complete. 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 Processor -- start 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 -- start start 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f1c101170 0x7f0f1c1050b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 0x7f0f1c103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c102d10 0x7f0f1c103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c10c480 con 0x7f0f1c102370 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c10c300 con 0x7f0f1c101170 2024-03-30T13:56:27.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f2410f700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1c10c600 con 0x7f0f1c102d10 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 0x7f0f1c103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 0x7f0f1c103750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36330/0 (socket says 172.21.15.102:36330) 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f21eab700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f1c101170 0x7f0f1c1050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f226ac700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c102d10 0x7f0f1c103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 -- 172.21.15.102:0/2377090004 learned_addr learned my addr 172.21.15.102:0/2377090004 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 -- 172.21.15.102:0/2377090004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c102d10 msgr2=0x7f0f1c103c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 --2- 172.21.15.102:0/2377090004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c102d10 0x7f0f1c103c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 -- 172.21.15.102:0/2377090004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f1c101170 msgr2=0x7f0f1c1050b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 --2- 172.21.15.102:0/2377090004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f1c101170 0x7f0f1c1050b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.490+0000 7f0f216aa700 1 -- 172.21.15.102:0/2377090004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c104560 con 0x7f0f1c102370 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.491+0000 7f0f216aa700 1 --2- 172.21.15.102:0/2377090004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 0x7f0f1c103750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800ec10 tx=0x7f0f1800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.491+0000 7f0f12ffd700 1 -- 172.21.15.102:0/2377090004 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1800cc90 con 0x7f0f1c102370 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.491+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1c1a1d80 con 0x7f0f1c102370 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.491+0000 7f0f12ffd700 1 -- 172.21.15.102:0/2377090004 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f0f1800ce30 con 0x7f0f1c102370 2024-03-30T13:56:27.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.491+0000 7f0f12ffd700 1 -- 172.21.15.102:0/2377090004 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f18010430 con 0x7f0f1c102370 2024-03-30T13:56:27.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.491+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f1c1a2180 con 0x7f0f1c102370 2024-03-30T13:56:27.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.492+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f1c068f00 con 0x7f0f1c102370 2024-03-30T13:56:27.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.493+0000 7f0f12ffd700 1 -- 172.21.15.102:0/2377090004 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f0f18003fa0 con 0x7f0f1c102370 2024-03-30T13:56:27.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.493+0000 7f0f12ffd700 1 --2- 172.21.15.102:0/2377090004 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0f0806db10 0x7f0f0806ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:27.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.493+0000 7f0f21eab700 1 --2- 172.21.15.102:0/2377090004 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0f0806db10 0x7f0f0806ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:27.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.493+0000 7f0f12ffd700 1 -- 172.21.15.102:0/2377090004 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 3941+0+0 (secure 0 0 0) 0x7f0f1808d560 con 0x7f0f1c102370 2024-03-30T13:56:27.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.493+0000 7f0f21eab700 1 --2- 172.21.15.102:0/2377090004 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0f0806db10 0x7f0f0806ffd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009fd0 tx=0x7f0f0c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:27.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.497+0000 7f0f12ffd700 1 -- 172.21.15.102:0/2377090004 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0f1805b9d0 con 0x7f0f1c102370 2024-03-30T13:56:27.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.690+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0f1c1a2590 con 0x7f0f1c102370 2024-03-30T13:56:27.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.690+0000 7f0f12ffd700 1 -- 172.21.15.102:0/2377090004 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v53) ==== 74+0+130 (secure 0 0 0) 0x7f0f1805b2d0 con 0x7f0f1c102370 2024-03-30T13:56:27.687 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:56:27.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0f0806db10 msgr2=0x7f0f0806ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:27.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 --2- 172.21.15.102:0/2377090004 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0f0806db10 0x7f0f0806ffd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009fd0 tx=0x7f0f0c019040 comp rx=0 tx=0).stop 2024-03-30T13:56:27.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 msgr2=0x7f0f1c103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:27.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 --2- 172.21.15.102:0/2377090004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 0x7f0f1c103750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800ec10 tx=0x7f0f1800ef20 comp rx=0 tx=0).stop 2024-03-30T13:56:27.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 shutdown_connections 2024-03-30T13:56:27.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 --2- 172.21.15.102:0/2377090004 >> v2:172.21.15.102:6800/3154847229 conn(0x7f0f0806db10 0x7f0f0806ffd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 --2- 172.21.15.102:0/2377090004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f1c102d10 0x7f0f1c103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 --2- 172.21.15.102:0/2377090004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f1c101170 0x7f0f1c1050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 --2- 172.21.15.102:0/2377090004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f1c102370 0x7f0f1c103750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 >> 172.21.15.102:0/2377090004 conn(0x7f0f1c0fc730 msgr2=0x7f0f1c06a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.692+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 shutdown_connections 2024-03-30T13:56:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:27.693+0000 7f0f2410f700 1 -- 172.21.15.102:0/2377090004 wait complete. 2024-03-30T13:56:27.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:27 smithi138 ceph-mon[33397]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:56:27.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:27 smithi138 ceph-mon[33397]: osdmap e53: 8 total, 7 up, 8 in 2024-03-30T13:56:27.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:27.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:27 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:27.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:27 smithi138 ceph-mon[33397]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:27.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2377090004' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[29392]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[29392]: osdmap e53: 8 total, 7 up, 8 in 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[29392]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2377090004' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[37717]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-03-30T13:56:27.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[37717]: osdmap e53: 8 total, 7 up, 8 in 2024-03-30T13:56:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[37717]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-30T13:56:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2377090004' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:56:28.477 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1711806963,"num_in_osds":8,"osd_in_since":1711806969,"num_remapped_pgs":0} 2024-03-30T13:56:28.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:28 smithi138 ceph-mon[33397]: purged_snaps scrub starts 2024-03-30T13:56:28.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:28 smithi138 ceph-mon[33397]: purged_snaps scrub ok 2024-03-30T13:56:28.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:28 smithi138 ceph-mon[33397]: osd.7 v2:172.21.15.138:6812/3765635388 boot 2024-03-30T13:56:28.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:28 smithi138 ceph-mon[33397]: osdmap e54: 8 total, 8 up, 8 in 2024-03-30T13:56:28.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:28 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:28.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:28 smithi138 ceph-mon[33397]: osdmap e55: 8 total, 8 up, 8 in 2024-03-30T13:56:29.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[29392]: purged_snaps scrub starts 2024-03-30T13:56:29.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[29392]: purged_snaps scrub ok 2024-03-30T13:56:29.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[29392]: osd.7 v2:172.21.15.138:6812/3765635388 boot 2024-03-30T13:56:29.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[29392]: osdmap e54: 8 total, 8 up, 8 in 2024-03-30T13:56:29.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:29.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[29392]: osdmap e55: 8 total, 8 up, 8 in 2024-03-30T13:56:29.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[37717]: purged_snaps scrub starts 2024-03-30T13:56:29.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[37717]: purged_snaps scrub ok 2024-03-30T13:56:29.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[37717]: osd.7 v2:172.21.15.138:6812/3765635388 boot 2024-03-30T13:56:29.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[37717]: osdmap e54: 8 total, 8 up, 8 in 2024-03-30T13:56:29.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:56:29.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:28 smithi102 ceph-mon[37717]: osdmap e55: 8 total, 8 up, 8 in 2024-03-30T13:56:29.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T13:56:29.722 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:30.145 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: Detected new or changed devices on smithi138 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi138 to 4272M 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:30.146 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[37717]: osdmap e56: 8 total, 8 up, 8 in 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: Detected new or changed devices on smithi138 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi138 to 4272M 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:30 smithi102 ceph-mon[29392]: osdmap e56: 8 total, 8 up, 8 in 2024-03-30T13:56:30.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: Detected new or changed devices on smithi138 2024-03-30T13:56:30.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi138 to 4272M 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:30.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:30 smithi138 ceph-mon[33397]: osdmap e56: 8 total, 8 up, 8 in 2024-03-30T13:56:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.295+0000 7f228ddf8700 1 -- 172.21.15.102:0/2055428039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288103490 msgr2=0x7f2288105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.295+0000 7f228ddf8700 1 --2- 172.21.15.102:0/2055428039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288103490 0x7f2288105920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f227c0098f0 tx=0x7f227c009c00 comp rx=0 tx=0).stop 2024-03-30T13:56:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.295+0000 7f228ddf8700 1 -- 172.21.15.102:0/2055428039 shutdown_connections 2024-03-30T13:56:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.295+0000 7f228ddf8700 1 --2- 172.21.15.102:0/2055428039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2288105e60 0x7f22881062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.295+0000 7f228ddf8700 1 --2- 172.21.15.102:0/2055428039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288100b20 0x7f2288102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.295+0000 7f228ddf8700 1 --2- 172.21.15.102:0/2055428039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288103490 0x7f2288105920 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 -- 172.21.15.102:0/2055428039 >> 172.21.15.102:0/2055428039 conn(0x7f22880fc760 msgr2=0x7f22880feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 -- 172.21.15.102:0/2055428039 shutdown_connections 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 -- 172.21.15.102:0/2055428039 wait complete. 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 Processor -- start 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 -- start start 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2288100b20 0x7f228819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288103490 0x7f228819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288105e60 0x7f22881a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2288113aa0 con 0x7f2288105e60 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2288113920 con 0x7f2288100b20 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.296+0000 7f228ddf8700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2288113c20 con 0x7f2288103490 2024-03-30T13:56:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288103490 0x7f228819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2287fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288105e60 0x7f22881a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288103490 0x7f228819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:60458/0 (socket says 172.21.15.102:60458) 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 -- 172.21.15.102:0/3319106872 learned_addr learned my addr 172.21.15.102:0/3319106872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f22877fe700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2288100b20 0x7f228819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 -- 172.21.15.102:0/3319106872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2288100b20 msgr2=0x7f228819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2288100b20 0x7f228819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 -- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288105e60 msgr2=0x7f22881a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288105e60 0x7f22881a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 -- 172.21.15.102:0/3319106872 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22881a5d70 con 0x7f2288103490 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2287fff700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288105e60 0x7f22881a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2286ffd700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288103490 0x7f228819e650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f227c009230 tx=0x7f227c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2284ff9700 1 -- 172.21.15.102:0/3319106872 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c00f840 con 0x7f2288103490 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22881a6000 con 0x7f2288103490 2024-03-30T13:56:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2284ff9700 1 -- 172.21.15.102:0/3319106872 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f227c005670 con 0x7f2288103490 2024-03-30T13:56:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f2284ff9700 1 -- 172.21.15.102:0/3319106872 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c017400 con 0x7f2288103490 2024-03-30T13:56:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.297+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f22881a64b0 con 0x7f2288103490 2024-03-30T13:56:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.298+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2288068f00 con 0x7f2288103490 2024-03-30T13:56:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.299+0000 7f2284ff9700 1 -- 172.21.15.102:0/3319106872 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f227c0046d0 con 0x7f2288103490 2024-03-30T13:56:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.299+0000 7f2284ff9700 1 --2- 172.21.15.102:0/3319106872 >> v2:172.21.15.102:6800/3154847229 conn(0x7f227006db10 0x7f227006ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.299+0000 7f22877fe700 1 --2- 172.21.15.102:0/3319106872 >> v2:172.21.15.102:6800/3154847229 conn(0x7f227006db10 0x7f227006ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.299+0000 7f2284ff9700 1 -- 172.21.15.102:0/3319106872 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 4081+0+0 (secure 0 0 0) 0x7f227c08e3b0 con 0x7f2288103490 2024-03-30T13:56:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.300+0000 7f22877fe700 1 --2- 172.21.15.102:0/3319106872 >> v2:172.21.15.102:6800/3154847229 conn(0x7f227006db10 0x7f227006ffd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2278009750 tx=0x7f2278009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.304+0000 7f2284ff9700 1 -- 172.21.15.102:0/3319106872 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f227c013030 con 0x7f2288103490 2024-03-30T13:56:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.498+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f22881a6b40 con 0x7f2288103490 2024-03-30T13:56:31.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.498+0000 7f2284ff9700 1 -- 172.21.15.102:0/3319106872 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v57) ==== 74+0+130 (secure 0 0 0) 0x7f227c05c710 con 0x7f2288103490 2024-03-30T13:56:31.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:56:31.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.500+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 >> v2:172.21.15.102:6800/3154847229 conn(0x7f227006db10 msgr2=0x7f227006ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:31.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.500+0000 7f228ddf8700 1 --2- 172.21.15.102:0/3319106872 >> v2:172.21.15.102:6800/3154847229 conn(0x7f227006db10 0x7f227006ffd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2278009750 tx=0x7f2278009310 comp rx=0 tx=0).stop 2024-03-30T13:56:31.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.500+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288103490 msgr2=0x7f228819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:31.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.500+0000 7f228ddf8700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288103490 0x7f228819e650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f227c009230 tx=0x7f227c005270 comp rx=0 tx=0).stop 2024-03-30T13:56:31.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 shutdown_connections 2024-03-30T13:56:31.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 --2- 172.21.15.102:0/3319106872 >> v2:172.21.15.102:6800/3154847229 conn(0x7f227006db10 0x7f227006ffd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2288105e60 0x7f22881a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2288100b20 0x7f228819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 --2- 172.21.15.102:0/3319106872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2288103490 0x7f228819e650 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:31.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 >> 172.21.15.102:0/3319106872 conn(0x7f22880fc760 msgr2=0x7f22881038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:31.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 shutdown_connections 2024-03-30T13:56:31.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:31.501+0000 7f228ddf8700 1 -- 172.21.15.102:0/3319106872 wait complete. 2024-03-30T13:56:31.862 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:31 smithi102 ceph-mon[37717]: osdmap e57: 8 total, 8 up, 8 in 2024-03-30T13:56:31.862 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:31 smithi102 ceph-mon[37717]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:31.862 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3319106872' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:56:31.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:31 smithi102 ceph-mon[29392]: osdmap e57: 8 total, 8 up, 8 in 2024-03-30T13:56:31.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:31 smithi102 ceph-mon[29392]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:31.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3319106872' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:56:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:31 smithi138 ceph-mon[33397]: osdmap e57: 8 total, 8 up, 8 in 2024-03-30T13:56:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:31 smithi138 ceph-mon[33397]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3319106872' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T13:56:32.211 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1711806987,"num_in_osds":8,"osd_in_since":1711806969,"num_remapped_pgs":0} 2024-03-30T13:56:32.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T13:56:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:32.823 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:32 smithi102 ceph-mon[29392]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T13:56:32.824 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:32 smithi102 ceph-mon[37717]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T13:56:33.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:32 smithi138 ceph-mon[33397]: osdmap e58: 8 total, 8 up, 8 in 2024-03-30T13:56:33.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:33 smithi102 ceph-mon[29392]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-30T13:56:33.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:33 smithi102 ceph-mon[37717]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-30T13:56:34.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:33 smithi138 ceph-mon[33397]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-30T13:56:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 -- 172.21.15.102:0/1756112065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 msgr2=0x7f7db41015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 --2- 172.21.15.102:0/1756112065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db41015a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009990 tx=0x7f7da8009ca0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 -- 172.21.15.102:0/1756112065 shutdown_connections 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 --2- 172.21.15.102:0/1756112065 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7db4101ae0 0x7f7db410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 --2- 172.21.15.102:0/1756112065 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7db4107180 0x7f7db4107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 --2- 172.21.15.102:0/1756112065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db41015a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 -- 172.21.15.102:0/1756112065 >> 172.21.15.102:0/1756112065 conn(0x7f7db4078070 msgr2=0x7f7db4078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 -- 172.21.15.102:0/1756112065 shutdown_connections 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 -- 172.21.15.102:0/1756112065 wait complete. 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 Processor -- start 2024-03-30T13:56:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.245+0000 7f7db9514700 1 -- start start 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db9514700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db4198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db9514700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7db4101ae0 0x7f7db4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db9514700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7db4107180 0x7f7db419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db9514700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7db4110b30 con 0x7f7db4101140 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db9514700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7db41109b0 con 0x7f7db4107180 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db9514700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7db4110cb0 con 0x7f7db4101ae0 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db4198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db27fc700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7db4101ae0 0x7f7db4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db4198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33440/0 (socket says 172.21.15.102:33440) 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db2ffd700 1 -- 172.21.15.102:0/1138715364 learned_addr learned my addr 172.21.15.102:0/1138715364 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db37fe700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7db4107180 0x7f7db419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db27fc700 1 -- 172.21.15.102:0/1138715364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7db4107180 msgr2=0x7f7db419ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db27fc700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7db4107180 0x7f7db419ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db27fc700 1 -- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 msgr2=0x7f7db4198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db27fc700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db4198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db27fc700 1 -- 172.21.15.102:0/1138715364 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db419d150 con 0x7f7db4101ae0 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db2ffd700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db4198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db27fc700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7db4101ae0 0x7f7db4198680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7da80098e0 tx=0x7f7da8005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7d9bfff700 1 -- 172.21.15.102:0/1138715364 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da8027070 con 0x7f7db4101ae0 2024-03-30T13:56:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7d9bfff700 1 -- 172.21.15.102:0/1138715364 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f7da80056b0 con 0x7f7db4101ae0 2024-03-30T13:56:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db419d3e0 con 0x7f7db4101ae0 2024-03-30T13:56:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.246+0000 7f7d9bfff700 1 -- 172.21.15.102:0/1138715364 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80173f0 con 0x7f7db4101ae0 2024-03-30T13:56:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.247+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db41a4cc0 con 0x7f7db4101ae0 2024-03-30T13:56:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.248+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db4068f00 con 0x7f7db4101ae0 2024-03-30T13:56:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.248+0000 7f7d9bfff700 1 -- 172.21.15.102:0/1138715364 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f7da802a070 con 0x7f7db4101ae0 2024-03-30T13:56:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.248+0000 7f7d9bfff700 1 --2- 172.21.15.102:0/1138715364 >> v2:172.21.15.102:6800/3154847229 conn(0x7f7d9c06db10 0x7f7d9c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.248+0000 7f7db2ffd700 1 --2- 172.21.15.102:0/1138715364 >> v2:172.21.15.102:6800/3154847229 conn(0x7f7d9c06db10 0x7f7d9c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.248+0000 7f7d9bfff700 1 -- 172.21.15.102:0/1138715364 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (secure 0 0 0) 0x7f7da808e1f0 con 0x7f7db4101ae0 2024-03-30T13:56:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.249+0000 7f7db2ffd700 1 --2- 172.21.15.102:0/1138715364 >> v2:172.21.15.102:6800/3154847229 conn(0x7f7d9c06db10 0x7f7d9c06ffd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7da40103d0 tx=0x7f7da4007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:34.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.253+0000 7f7d9bfff700 1 -- 172.21.15.102:0/1138715364 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7da8013070 con 0x7f7db4101ae0 2024-03-30T13:56:34.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.448+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7db41a4e90 con 0x7f7db4101ae0 2024-03-30T13:56:34.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.449+0000 7f7d9bfff700 1 -- 172.21.15.102:0/1138715364 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+11543 (secure 0 0 0) 0x7f7da805c550 con 0x7f7db4101ae0 2024-03-30T13:56:34.446 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:56:34.446 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":58,"fsid":"69dd7ec8-ee9c-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:51:13.147429+0000","modified":"2024-03-30T13:56:31.726643+0000","last_up_change":"2024-03-30T13:56:27.720377+0000","last_in_change":"2024-03-30T13:56:09.983286+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:54:25.314887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d9e04b0b-a4df-4d4f-87f8-59e29edd2e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6801","nonce":3100565996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3100565996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3100565996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6803","nonce":3100565996}]},"public_addr":"172.21.15.102:6801/3100565996","cluster_addr":"172.21.15.102:6802/3100565996","heartbeat_back_addr":"172.21.15.102:6804/3100565996","heartbeat_front_addr":"172.21.15.102:6803/3100565996","state":["exists","up"]},{"osd":1,"uuid":"046093f3-4186-496a-a5b2-9e8d4b86a2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":3593563159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3593563159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3593563159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6807","nonce":3593563159}]},"public_addr":"172.21.15.102:6805/3593563159","cluster_addr":"172.21.15.102:6806/3593563159","heartbeat_back_addr":"172.21.15.102:6808/3593563159","heartbeat_front_addr":"172.21.15.102:6807/3593563159","state":["exists","up"]},{"osd":2,"uuid":"4779686a-2976-4755-9116-8835f28fa5a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":203063794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":203063794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":203063794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6811","nonce":203063794}]},"public_addr":"172.21.15.102:6809/203063794","cluster_addr":"172.21.15.102:6810/203063794","heartbeat_back_addr":"172.21.15.102:6812/203063794","heartbeat_front_addr":"172.21.15.102:6811/203063794","state":["exists","up"]},{"osd":3,"uuid":"2cd657e0-4c72-45db-a26b-d86c06d2996b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":1080330048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1080330048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1080330048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6815","nonce":1080330048}]},"public_addr":"172.21.15.102:6813/1080330048","cluster_addr":"172.21.15.102:6814/1080330048","heartbeat_back_addr":"172.21.15.102:6816/1080330048","heartbeat_front_addr":"172.21.15.102:6815/1080330048","state":["exists","up"]},{"osd":4,"uuid":"a69453af-c657-49a7-b759-bebd7c7fefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":1718735591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":1718735591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":1718735591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":1718735591}]},"public_addr":"172.21.15.138:6800/1718735591","cluster_addr":"172.21.15.138:6801/1718735591","heartbeat_back_addr":"172.21.15.138:6803/1718735591","heartbeat_front_addr":"172.21.15.138:6802/1718735591","state":["exists","up"]},{"osd":5,"uuid":"2b723775-3bc5-4c26-8c7b-570cfe9a270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":1568298628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":1568298628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":1568298628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":1568298628}]},"public_addr":"172.21.15.138:6804/1568298628","cluster_addr":"172.21.15.138:6805/1568298628","heartbeat_back_addr":"172.21.15.138:6807/1568298628","heartbeat_front_addr":"172.21.15.138:6806/1568298628","state":["exists","up"]},{"osd":6,"uuid":"9cdfc0c0-2f91-44d9-9c0a-401e328da8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3214961690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":3214961690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":3214961690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3214961690}]},"public_addr":"172.21.15.138:6808/3214961690","cluster_addr":"172.21.15.138:6809/3214961690","heartbeat_back_addr":"172.21.15.138:6811/3214961690","heartbeat_front_addr":"172.21.15.138:6810/3214961690","state":["exists","up"]},{"osd":7,"uuid":"67ed0818-2486-420e-97ff-0264fac6a48b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3765635388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":3765635388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":3765635388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3765635388}]},"public_addr":"172.21.15.138:6812/3765635388","cluster_addr":"172.21.15.138:6813/3765635388","heartbeat_back_addr":"172.21.15.138:6815/3765635388","heartbeat_front_addr":"172.21.15.138:6814/3765635388","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:53:30.706660+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:53:55.797625+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:54:21.798206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:54:48.334156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:55:11.795826+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:55:36.167708+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:56:01.269021+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:56:25.786109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1457123288":"2024-03-31T13:52:06.991127+0000","172.21.15.102:0/1446807484":"2024-03-31T13:52:06.991127+0000","172.21.15.102:0/10326931":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/2083482926":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/1947582370":"2024-03-31T13:51:40.671168+0000","172.21.15.102:6800/2520146638":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/335601191":"2024-03-31T13:52:06.991127+0000","172.21.15.102:6800/3737793096":"2024-03-31T13:52:06.991127+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 >> v2:172.21.15.102:6800/3154847229 conn(0x7f7d9c06db10 msgr2=0x7f7d9c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 --2- 172.21.15.102:0/1138715364 >> v2:172.21.15.102:6800/3154847229 conn(0x7f7d9c06db10 0x7f7d9c06ffd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7da40103d0 tx=0x7f7da4007480 comp rx=0 tx=0).stop 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7db4101ae0 msgr2=0x7f7db4198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7db4101ae0 0x7f7db4198680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7da80098e0 tx=0x7f7da8005590 comp rx=0 tx=0).stop 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 shutdown_connections 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 --2- 172.21.15.102:0/1138715364 >> v2:172.21.15.102:6800/3154847229 conn(0x7f7d9c06db10 0x7f7d9c06ffd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db4101140 0x7f7db4198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7db4107180 0x7f7db419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 --2- 172.21.15.102:0/1138715364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7db4101ae0 0x7f7db4198680 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:34.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 >> 172.21.15.102:0/1138715364 conn(0x7f7db4078070 msgr2=0x7f7db4069ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:34.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 shutdown_connections 2024-03-30T13:56:34.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:34.451+0000 7f7db9514700 1 -- 172.21.15.102:0/1138715364 wait complete. 2024-03-30T13:56:35.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1138715364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:56:35.075 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1138715364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:56:35.101 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T13:54:25.314887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T13:56:35.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-30T13:56:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1138715364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:56:35.339 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:56:36.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:35 smithi138 ceph-mon[33397]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:56:36.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:35 smithi102 ceph-mon[29392]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:56:36.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:35 smithi102 ceph-mon[37717]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T13:56:36.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 -- 172.21.15.102:0/2597909867 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101140 msgr2=0x7f6c101015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/2597909867 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101140 0x7f6c101015a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009990 tx=0x7f6c00009ca0 comp rx=0 tx=0).stop 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 -- 172.21.15.102:0/2597909867 shutdown_connections 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/2597909867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c10101ae0 0x7f6c1010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/2597909867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c10107180 0x7f6c10107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/2597909867 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101140 0x7f6c101015a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 -- 172.21.15.102:0/2597909867 >> 172.21.15.102:0/2597909867 conn(0x7f6c10078070 msgr2=0x7f6c10078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 -- 172.21.15.102:0/2597909867 shutdown_connections 2024-03-30T13:56:36.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.983+0000 7f6c14d3f700 1 -- 172.21.15.102:0/2597909867 wait complete. 2024-03-30T13:56:36.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 Processor -- start 2024-03-30T13:56:36.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 -- start start 2024-03-30T13:56:36.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c10101140 0x7f6c10198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:36.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101ae0 0x7f6c101993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c10107180 0x7f6c101a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10110af0 con 0x7f6c10101140 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6c10110970 con 0x7f6c10107180 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c14d3f700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6c10110c70 con 0x7f6c10101ae0 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c0dd9b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101ae0 0x7f6c101993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c0dd9b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101ae0 0x7f6c101993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:60492/0 (socket says 172.21.15.102:60492) 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c0dd9b700 1 -- 172.21.15.102:0/3863223579 learned_addr learned my addr 172.21.15.102:0/3863223579 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c0e59c700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c10101140 0x7f6c10198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c0dd9b700 1 -- 172.21.15.102:0/3863223579 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c10107180 msgr2=0x7f6c101a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.984+0000 7f6c0ed9d700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c10107180 0x7f6c101a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6c0dd9b700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c10107180 0x7f6c101a3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6c0dd9b700 1 -- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c10101140 msgr2=0x7f6c10198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6c0dd9b700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c10101140 0x7f6c10198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6c0dd9b700 1 -- 172.21.15.102:0/3863223579 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c101a4460 con 0x7f6c10101ae0 2024-03-30T13:56:36.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6c0dd9b700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101ae0 0x7f6c101993e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c000057f0 tx=0x7f6c0000bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6bff7fe700 1 -- 172.21.15.102:0/3863223579 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c00026070 con 0x7f6c10101ae0 2024-03-30T13:56:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6bff7fe700 1 -- 172.21.15.102:0/3863223579 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f6c00021410 con 0x7f6c10101ae0 2024-03-30T13:56:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c101a4630 con 0x7f6c10101ae0 2024-03-30T13:56:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6bff7fe700 1 -- 172.21.15.102:0/3863223579 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c00017650 con 0x7f6c10101ae0 2024-03-30T13:56:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.985+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c101a4b30 con 0x7f6c10101ae0 2024-03-30T13:56:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.986+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c10068f00 con 0x7f6c10101ae0 2024-03-30T13:56:36.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.986+0000 7f6bff7fe700 1 -- 172.21.15.102:0/3863223579 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f6c00029070 con 0x7f6c10101ae0 2024-03-30T13:56:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.987+0000 7f6bff7fe700 1 --2- 172.21.15.102:0/3863223579 >> v2:172.21.15.102:6800/3154847229 conn(0x7f6bf806db10 0x7f6bf806ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.987+0000 7f6bff7fe700 1 -- 172.21.15.102:0/3863223579 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (secure 0 0 0) 0x7f6c0008e010 con 0x7f6c10101ae0 2024-03-30T13:56:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.987+0000 7f6c0e59c700 1 --2- 172.21.15.102:0/3863223579 >> v2:172.21.15.102:6800/3154847229 conn(0x7f6bf806db10 0x7f6bf806ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:36.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.987+0000 7f6c0e59c700 1 --2- 172.21.15.102:0/3863223579 >> v2:172.21.15.102:6800/3154847229 conn(0x7f6bf806db10 0x7f6bf806ffd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400abd0 tx=0x7f6c0400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:36.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:36.991+0000 7f6bff7fe700 1 -- 172.21.15.102:0/3863223579 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c00013070 con 0x7f6c10101ae0 2024-03-30T13:56:37.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.188+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6c101a4e50 con 0x7f6c10101ae0 2024-03-30T13:56:37.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.189+0000 7f6bff7fe700 1 -- 172.21.15.102:0/3863223579 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (secure 0 0 0) 0x7f6c0005c3f0 con 0x7f6c10101ae0 2024-03-30T13:56:37.186 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-30T13:56:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 >> v2:172.21.15.102:6800/3154847229 conn(0x7f6bf806db10 msgr2=0x7f6bf806ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/3863223579 >> v2:172.21.15.102:6800/3154847229 conn(0x7f6bf806db10 0x7f6bf806ffd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400abd0 tx=0x7f6c0400a380 comp rx=0 tx=0).stop 2024-03-30T13:56:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101ae0 msgr2=0x7f6c101993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101ae0 0x7f6c101993e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c000057f0 tx=0x7f6c0000bef0 comp rx=0 tx=0).stop 2024-03-30T13:56:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 shutdown_connections 2024-03-30T13:56:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/3863223579 >> v2:172.21.15.102:6800/3154847229 conn(0x7f6bf806db10 0x7f6bf806ffd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:37.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c10101140 0x7f6c10198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:37.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c10107180 0x7f6c101a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:37.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 --2- 172.21.15.102:0/3863223579 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6c10101ae0 0x7f6c101993e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:37.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 >> 172.21.15.102:0/3863223579 conn(0x7f6c10078070 msgr2=0x7f6c10104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:37.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 shutdown_connections 2024-03-30T13:56:37.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:56:37.191+0000 7f6c14d3f700 1 -- 172.21.15.102:0/3863223579 wait complete. 2024-03-30T13:56:37.902 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi102 2024-03-30T13:56:37.903 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply rgw foo.a --placement '1;smithi102=foo.a' 2024-03-30T13:56:38.155 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:38.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:38 smithi138 ceph-mon[33397]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-30T13:56:38.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3863223579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T13:56:38.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:38 smithi102 ceph-mon[29392]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-30T13:56:38.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3863223579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T13:56:38.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:38 smithi102 ceph-mon[37717]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-30T13:56:38.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3863223579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T13:56:39.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.652+0000 7faba2ad4700 1 -- 172.21.15.138:0/1001148639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c1023b0 msgr2=0x7fab9c102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.652+0000 7faba2ad4700 1 --2- 172.21.15.138:0/1001148639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c1023b0 0x7fab9c102810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fab880099c0 tx=0x7fab88009cd0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.652+0000 7faba2ad4700 1 -- 172.21.15.138:0/1001148639 shutdown_connections 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.652+0000 7faba2ad4700 1 --2- 172.21.15.138:0/1001148639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c102d50 0x7fab9c109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.652+0000 7faba2ad4700 1 --2- 172.21.15.138:0/1001148639 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab9c1011b0 0x7fab9c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.652+0000 7faba2ad4700 1 --2- 172.21.15.138:0/1001148639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c1023b0 0x7fab9c102810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.652+0000 7faba2ad4700 1 -- 172.21.15.138:0/1001148639 >> 172.21.15.138:0/1001148639 conn(0x7fab9c0fc730 msgr2=0x7fab9c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 -- 172.21.15.138:0/1001148639 shutdown_connections 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 -- 172.21.15.138:0/1001148639 wait complete. 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 Processor -- start 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 -- start start 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab9c1011b0 0x7fab9c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c1023b0 0x7fab9c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c102d50 0x7fab9c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:39.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10c4c0 con 0x7fab9c1023b0 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10c340 con 0x7fab9c102d50 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba2ad4700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c10c640 con 0x7fab9c1011b0 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba1071700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c102d50 0x7fab9c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba1071700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c102d50 0x7fab9c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46802/0 (socket says 172.21.15.138:46802) 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.653+0000 7faba1071700 1 -- 172.21.15.138:0/3013095823 learned_addr learned my addr 172.21.15.138:0/3013095823 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba0870700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab9c1011b0 0x7fab9c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7fab9bfff700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c1023b0 0x7fab9c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba1071700 1 -- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab9c1011b0 msgr2=0x7fab9c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba1071700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab9c1011b0 0x7fab9c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba1071700 1 -- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c1023b0 msgr2=0x7fab9c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba1071700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c1023b0 0x7fab9c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba1071700 1 -- 172.21.15.138:0/3013095823 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c19f6e0 con 0x7fab9c102d50 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba1071700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c102d50 0x7fab9c19f040 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fab94007d70 tx=0x7fab9400a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:39.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7fab99ffb700 1 -- 172.21.15.138:0/3013095823 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94016020 con 0x7fab9c102d50 2024-03-30T13:56:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7fab99ffb700 1 -- 172.21.15.138:0/3013095823 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fab9400ad40 con 0x7fab9c102d50 2024-03-30T13:56:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c19f9d0 con 0x7fab9c102d50 2024-03-30T13:56:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7fab99ffb700 1 -- 172.21.15.138:0/3013095823 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab940115e0 con 0x7fab9c102d50 2024-03-30T13:56:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.654+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c19ff60 con 0x7fab9c102d50 2024-03-30T13:56:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.655+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c068f00 con 0x7fab9c102d50 2024-03-30T13:56:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.656+0000 7fab99ffb700 1 -- 172.21.15.138:0/3013095823 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7fab94019020 con 0x7fab9c102d50 2024-03-30T13:56:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.656+0000 7fab99ffb700 1 --2- 172.21.15.138:0/3013095823 >> v2:172.21.15.102:6800/3154847229 conn(0x7fab8c06db10 0x7fab8c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.656+0000 7fab99ffb700 1 -- 172.21.15.138:0/3013095823 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (secure 0 0 0) 0x7fab9408d130 con 0x7fab9c102d50 2024-03-30T13:56:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.656+0000 7faba0870700 1 --2- 172.21.15.138:0/3013095823 >> v2:172.21.15.102:6800/3154847229 conn(0x7fab8c06db10 0x7fab8c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.656+0000 7faba0870700 1 --2- 172.21.15.138:0/3013095823 >> v2:172.21.15.102:6800/3154847229 conn(0x7fab8c06db10 0x7fab8c06ffd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fab90005fd0 tx=0x7fab90005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.661+0000 7fab99ffb700 1 -- 172.21.15.138:0/3013095823 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fab94020020 con 0x7fab9c102d50 2024-03-30T13:56:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.863+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}) -- 0x7fab9c1a01f0 con 0x7fab8c06db10 2024-03-30T13:56:39.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.870+0000 7fab99ffb700 1 -- 172.21.15.138:0/3013095823 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7fab9c1a01f0 con 0x7fab8c06db10 2024-03-30T13:56:39.867 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled rgw.foo.a update... 2024-03-30T13:56:39.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 >> v2:172.21.15.102:6800/3154847229 conn(0x7fab8c06db10 msgr2=0x7fab8c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:39.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 --2- 172.21.15.138:0/3013095823 >> v2:172.21.15.102:6800/3154847229 conn(0x7fab8c06db10 0x7fab8c06ffd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fab90005fd0 tx=0x7fab90005e50 comp rx=0 tx=0).stop 2024-03-30T13:56:39.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c102d50 msgr2=0x7fab9c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:39.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c102d50 0x7fab9c19f040 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fab94007d70 tx=0x7fab9400a340 comp rx=0 tx=0).stop 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 shutdown_connections 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 --2- 172.21.15.138:0/3013095823 >> v2:172.21.15.102:6800/3154847229 conn(0x7fab8c06db10 0x7fab8c06ffd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab9c102d50 0x7fab9c19f040 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab9c1011b0 0x7fab9c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 --2- 172.21.15.138:0/3013095823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c1023b0 0x7fab9c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.872+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 >> 172.21.15.138:0/3013095823 conn(0x7fab9c0fc730 msgr2=0x7fab9c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.873+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 shutdown_connections 2024-03-30T13:56:39.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:39.873+0000 7faba2ad4700 1 -- 172.21.15.138:0/3013095823 wait complete. 2024-03-30T13:56:40.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:40 smithi138 ceph-mon[33397]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T13:56:40.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:40 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:40.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:40 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:40.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:40 smithi102 ceph-mon[29392]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T13:56:40.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:40 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:40.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:40 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:40.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:40 smithi102 ceph-mon[37717]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T13:56:40.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:40 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:40.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:40 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:40.486 DEBUG:teuthology.orchestra.run.smithi102:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@rgw.foo.a.service 2024-03-30T13:56:40.488 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi138 2024-03-30T13:56:40.488 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd pool create datapool 3 3 replicated 2024-03-30T13:56:40.724 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:41.329 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:41.330 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-30T13:56:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T13:56:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T13:56:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:41.331 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:41 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:41.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:41.448 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-30T13:56:41.448 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:41.448 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:41.448 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:41.448 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T13:56:41.448 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T13:56:41.449 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:41.449 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:41 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:42.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 -- 172.21.15.138:0/2569710129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 msgr2=0x7fd8401027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 --2- 172.21.15.138:0/2569710129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd8401027d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd8300099c0 tx=0x7fd830009cd0 comp rx=0 tx=0).stop 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 -- 172.21.15.138:0/2569710129 shutdown_connections 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 --2- 172.21.15.138:0/2569710129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd840101170 0x7fd840101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 --2- 172.21.15.138:0/2569710129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd840102d10 0x7fd8401095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 --2- 172.21.15.138:0/2569710129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd8401027d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 -- 172.21.15.138:0/2569710129 >> 172.21.15.138:0/2569710129 conn(0x7fd8400fc710 msgr2=0x7fd8400feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 -- 172.21.15.138:0/2569710129 shutdown_connections 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.250+0000 7fd847ce7700 1 -- 172.21.15.138:0/2569710129 wait complete. 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 Processor -- start 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 -- start start 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd840101170 0x7fd840197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd840198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:42.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd840102d10 0x7fd84019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd84010c4a0 con 0x7fd840101170 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd84010c320 con 0x7fd840102370 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd847ce7700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd84010c620 con 0x7fd840102d10 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd840198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd840198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46820/0 (socket says 172.21.15.138:46820) 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 -- 172.21.15.138:0/1463828241 learned_addr learned my addr 172.21.15.138:0/1463828241 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845a83700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd840101170 0x7fd840197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd846284700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd840102d10 0x7fd84019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 -- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd840102d10 msgr2=0x7fd84019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd840102d10 0x7fd84019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 -- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd840101170 msgr2=0x7fd840197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:42.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd840101170 0x7fd840197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:42.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 -- 172.21.15.138:0/1463828241 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84019f830 con 0x7fd840102370 2024-03-30T13:56:42.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd845282700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd840198160 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd830006010 tx=0x7fd8300047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:42.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd836ffd700 1 -- 172.21.15.138:0/1463828241 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830027070 con 0x7fd840102370 2024-03-30T13:56:42.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.251+0000 7fd836ffd700 1 -- 172.21.15.138:0/1463828241 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fd8300040c0 con 0x7fd840102370 2024-03-30T13:56:42.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.252+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84019fac0 con 0x7fd840102370 2024-03-30T13:56:42.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.252+0000 7fd836ffd700 1 -- 172.21.15.138:0/1463828241 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83000f460 con 0x7fd840102370 2024-03-30T13:56:42.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.252+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd84019ff10 con 0x7fd840102370 2024-03-30T13:56:42.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.252+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd840068f00 con 0x7fd840102370 2024-03-30T13:56:42.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.253+0000 7fd836ffd700 1 -- 172.21.15.138:0/1463828241 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7fd83002a070 con 0x7fd840102370 2024-03-30T13:56:42.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.253+0000 7fd836ffd700 1 --2- 172.21.15.138:0/1463828241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fd82c06db10 0x7fd82c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:42.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.254+0000 7fd836ffd700 1 -- 172.21.15.138:0/1463828241 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (secure 0 0 0) 0x7fd83008e200 con 0x7fd840102370 2024-03-30T13:56:42.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.254+0000 7fd845a83700 1 --2- 172.21.15.138:0/1463828241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fd82c06db10 0x7fd82c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:42.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.254+0000 7fd845a83700 1 --2- 172.21.15.138:0/1463828241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fd82c06db10 0x7fd82c06ffd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd83c005950 tx=0x7fd83c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:42.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.258+0000 7fd836ffd700 1 -- 172.21.15.138:0/1463828241 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd830013070 con 0x7fd840102370 2024-03-30T13:56:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:42 smithi102 ceph-mon[29392]: Deploying daemon rgw.foo.a on smithi102 2024-03-30T13:56:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:42 smithi102 ceph-mon[29392]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:56:42.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:42 smithi102 ceph-mon[37717]: Deploying daemon rgw.foo.a on smithi102 2024-03-30T13:56:42.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:42 smithi102 ceph-mon[37717]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:56:42.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:42 smithi138 ceph-mon[33397]: Deploying daemon rgw.foo.a on smithi102 2024-03-30T13:56:42.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:42 smithi138 ceph-mon[33397]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T13:56:42.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:42.450+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7fd8401a00e0 con 0x7fd840102370 2024-03-30T13:56:42.689 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 13:56:42 smithi102 systemd[1]: Starting Ceph rgw.foo.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:56:43.060 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 13:56:42 smithi102 podman[64291]: 2024-03-30 13:56:42.695281009 +0000 UTC m=+0.018709431 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:56:43.061 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 13:56:42 smithi102 podman[64291]: 2024-03-30 13:56:42.803444172 +0000 UTC m=+0.126872578 container create be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc) 2024-03-30T13:56:43.061 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 13:56:43 smithi102 podman[64291]: 2024-03-30 13:56:43.064541797 +0000 UTC m=+0.387970220 container init be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326) 2024-03-30T13:56:43.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.104+0000 7fd836ffd700 1 -- 172.21.15.138:0/1463828241 <== mon.1 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) ==== 160+0+0 (secure 0 0 0) 0x7fd83008e020 con 0x7fd840102370 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fd82c06db10 msgr2=0x7fd82c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 --2- 172.21.15.138:0/1463828241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fd82c06db10 0x7fd82c06ffd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd83c005950 tx=0x7fd83c0058e0 comp rx=0 tx=0).stop 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 msgr2=0x7fd840198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd840198160 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd830006010 tx=0x7fd8300047b0 comp rx=0 tx=0).stop 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 shutdown_connections 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 --2- 172.21.15.138:0/1463828241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fd82c06db10 0x7fd82c06ffd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd840101170 0x7fd840197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd840102d10 0x7fd84019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 --2- 172.21.15.138:0/1463828241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd840102370 0x7fd840198160 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 >> 172.21.15.138:0/1463828241 conn(0x7fd8400fc710 msgr2=0x7fd840069ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 shutdown_connections 2024-03-30T13:56:43.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:43.107+0000 7fd847ce7700 1 -- 172.21.15.138:0/1463828241 wait complete. 2024-03-30T13:56:43.105 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'datapool' created 2024-03-30T13:56:43.405 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/1463828241' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:56:43.405 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:56:43.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/1463828241' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:56:43.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:56:43.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/1463828241' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:56:43.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T13:56:43.439 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 13:56:43 smithi102 podman[64291]: 2024-03-30 13:56:43.067138248 +0000 UTC m=+0.390566663 container start be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-30T13:56:43.439 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 13:56:43 smithi102 bash[64291]: be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 2024-03-30T13:56:43.439 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 13:56:43 smithi102 systemd[1]: Started Ceph rgw.foo.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:56:43.721 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- rbd pool init datapool 2024-03-30T13:56:43.961 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:44.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-30T13:56:44.376 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T13:56:44.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T13:56:44.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.377 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:44 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:44.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-30T13:56:44.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T13:56:44.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T13:56:44.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: osdmap e59: 8 total, 8 up, 8 in 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:44.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:44 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:45.408 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-30T13:56:45.408 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T13:56:45.408 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3078737918' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:56:45.408 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:45.408 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:45.408 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:45.409 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:45.409 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3078737918' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T13:56:45.409 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:45 smithi138 ceph-mon[33397]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3078737918' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3078737918' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T13:56:45.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[37717]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: osdmap e60: 8 total, 8 up, 8 in 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3078737918' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3078737918' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T13:56:45.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:45 smithi102 ceph-mon[29392]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T13:56:46.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: pgmap v189: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:46.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: Checking dashboard <-> RGW credentials 2024-03-30T13:56:46.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/3495155625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:56:46.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:56:46.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:56:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T13:56:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T13:56:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:46 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: pgmap v189: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: Checking dashboard <-> RGW credentials 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/3495155625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: pgmap v189: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: Checking dashboard <-> RGW credentials 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/3495155625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:46.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:46 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T13:56:48.411 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:48 smithi138 ceph-mon[33397]: pgmap v192: 68 pgs: 7 creating+peering, 8 active+clean, 53 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:48.412 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:48 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:48.412 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:56:48.412 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:48 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:56:48.412 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:48 smithi138 ceph-mon[33397]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T13:56:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[29392]: pgmap v192: 68 pgs: 7 creating+peering, 8 active+clean, 53 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:56:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:56:48.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[29392]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T13:56:48.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[37717]: pgmap v192: 68 pgs: 7 creating+peering, 8 active+clean, 53 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:56:48.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:48.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:56:48.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T13:56:48.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:48 smithi102 ceph-mon[37717]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T13:56:48.761 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.138 --placement '1;smithi138=iscsi.a' 2024-03-30T13:56:49.002 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:49.416 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:49 smithi138 ceph-mon[33397]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T13:56:49.416 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.416 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.416 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.416 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:49 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[37717]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[29392]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:49.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:49 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T13:56:50.425 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:50 smithi138 ceph-mon[33397]: pgmap v195: 100 pgs: 2 creating+activating, 18 creating+peering, 58 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:56:50.425 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:56:50.425 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:50 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:56:50.425 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:50 smithi138 ceph-mon[33397]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[29392]: pgmap v195: 100 pgs: 2 creating+activating, 18 creating+peering, 58 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[29392]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[37717]: pgmap v195: 100 pgs: 2 creating+activating, 18 creating+peering, 58 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T13:56:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:50 smithi102 ceph-mon[37717]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T13:56:50.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 -- 172.21.15.138:0/3167131202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900102d10 msgr2=0x7f7900105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:50.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 --2- 172.21.15.138:0/3167131202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900102d10 0x7f7900105100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009990 tx=0x7f78e8009ca0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 -- 172.21.15.138:0/3167131202 shutdown_connections 2024-03-30T13:56:50.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 --2- 172.21.15.138:0/3167131202 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7900107f70 0x7f790010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 --2- 172.21.15.138:0/3167131202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900102d10 0x7f7900105100 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 --2- 172.21.15.138:0/3167131202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900105640 0x7f7900107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 -- 172.21.15.138:0/3167131202 >> 172.21.15.138:0/3167131202 conn(0x7f79000fc740 msgr2=0x7f79000feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:50.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 -- 172.21.15.138:0/3167131202 shutdown_connections 2024-03-30T13:56:50.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.463+0000 7f7904fdc700 1 -- 172.21.15.138:0/3167131202 wait complete. 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 Processor -- start 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 -- start start 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900102d10 0x7f79001979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7900105640 0x7f7900197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900107f70 0x7f790019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f790010d560 con 0x7f7900102d10 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f790010d3e0 con 0x7f7900107f70 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f7904fdc700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f790010d6e0 con 0x7f7900105640 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f78fed9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900107f70 0x7f790019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f78fe59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900102d10 0x7f79001979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:50.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f78fe59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900102d10 0x7f79001979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.138:51434/0 (socket says 172.21.15.138:51434) 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f78fed9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900107f70 0x7f790019ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43394/0 (socket says 172.21.15.138:43394) 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f78fe59c700 1 -- 172.21.15.138:0/2119723206 learned_addr learned my addr 172.21.15.138:0/2119723206 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.464+0000 7f78fdd9b700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7900105640 0x7f7900197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78fed9d700 1 -- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7900105640 msgr2=0x7f7900197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78fed9d700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7900105640 0x7f7900197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78fed9d700 1 -- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900102d10 msgr2=0x7f79001979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78fed9d700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900102d10 0x7f79001979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78fed9d700 1 -- 172.21.15.138:0/2119723206 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790019f5f0 con 0x7f7900107f70 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78fe59c700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900102d10 0x7f79001979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78fed9d700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900107f70 0x7f790019ef50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f78f800cc20 tx=0x7f78f800cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78f77fe700 1 -- 172.21.15.138:0/2119723206 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8016020 con 0x7f7900107f70 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78f77fe700 1 -- 172.21.15.138:0/2119723206 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f78f8012420 con 0x7f7900107f70 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7900100b60 con 0x7f7900107f70 2024-03-30T13:56:50.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f78f77fe700 1 -- 172.21.15.138:0/2119723206 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f80115c0 con 0x7f7900107f70 2024-03-30T13:56:50.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.465+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7900101050 con 0x7f7900107f70 2024-03-30T13:56:50.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.466+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7900050750 con 0x7f7900107f70 2024-03-30T13:56:50.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.467+0000 7f78f77fe700 1 -- 172.21.15.138:0/2119723206 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f78f8019020 con 0x7f7900107f70 2024-03-30T13:56:50.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.467+0000 7f78f77fe700 1 --2- 172.21.15.138:0/2119723206 >> v2:172.21.15.102:6800/3154847229 conn(0x7f78ec06dbb0 0x7f78ec070070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:50.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.467+0000 7f78f77fe700 1 -- 172.21.15.138:0/2119723206 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5915+0+0 (secure 0 0 0) 0x7f78f808d6e0 con 0x7f7900107f70 2024-03-30T13:56:50.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.468+0000 7f78fe59c700 1 --2- 172.21.15.138:0/2119723206 >> v2:172.21.15.102:6800/3154847229 conn(0x7f78ec06dbb0 0x7f78ec070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:50.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.468+0000 7f78fe59c700 1 --2- 172.21.15.138:0/2119723206 >> v2:172.21.15.102:6800/3154847229 conn(0x7f78ec06dbb0 0x7f78ec070070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009750 tx=0x7f78e8005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:50.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.472+0000 7f78f77fe700 1 -- 172.21.15.138:0/2119723206 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f78f8020020 con 0x7f7900107f70 2024-03-30T13:56:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.690+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7f7900109af0 con 0x7f78ec06dbb0 2024-03-30T13:56:50.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.697+0000 7f78f77fe700 1 -- 172.21.15.138:0/2119723206 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (secure 0 0 0) 0x7f7900109af0 con 0x7f78ec06dbb0 2024-03-30T13:56:50.694 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled iscsi.datapool update... 2024-03-30T13:56:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 >> v2:172.21.15.102:6800/3154847229 conn(0x7f78ec06dbb0 msgr2=0x7f78ec070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 --2- 172.21.15.138:0/2119723206 >> v2:172.21.15.102:6800/3154847229 conn(0x7f78ec06dbb0 0x7f78ec070070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009750 tx=0x7f78e8005bc0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900107f70 msgr2=0x7f790019ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900107f70 0x7f790019ef50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f78f800cc20 tx=0x7f78f800cf30 comp rx=0 tx=0).stop 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 shutdown_connections 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 --2- 172.21.15.138:0/2119723206 >> v2:172.21.15.102:6800/3154847229 conn(0x7f78ec06dbb0 0x7f78ec070070 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7900102d10 0x7f79001979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7900107f70 0x7f790019ef50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 --2- 172.21.15.138:0/2119723206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7900105640 0x7f7900197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.699+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 >> 172.21.15.138:0/2119723206 conn(0x7f79000fc740 msgr2=0x7f79001083d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.700+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 shutdown_connections 2024-03-30T13:56:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:50.700+0000 7f7904fdc700 1 -- 172.21.15.138:0/2119723206 wait complete. 2024-03-30T13:56:51.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:51 smithi138 ceph-mon[33397]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T13:56:51.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:51 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:51 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:51.327 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-30T13:56:51.327 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:56:51.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-30T13:56:51.360 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:56:51.361 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-30T13:56:51.393 DEBUG:teuthology.orchestra.run.smithi138:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service 2024-03-30T13:56:51.436 INFO:tasks.cephadm:Adding prometheus.a on smithi138 2024-03-30T13:56:51.436 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply prometheus '1;smithi138=a' 2024-03-30T13:56:51.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[29392]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T13:56:51.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:51.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[37717]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T13:56:51.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T13:56:51.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:51 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:51.713 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: Saving service iscsi.datapool spec with placement smithi138=iscsi.a;count:1 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: pgmap v198: 132 pgs: 2 creating+activating, 18 creating+peering, 64 active+clean, 48 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: Saving service iscsi.datapool spec with placement smithi138=iscsi.a;count:1 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: pgmap v198: 132 pgs: 2 creating+activating, 18 creating+peering, 64 active+clean, 48 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.138", "placement": "1;smithi138=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:52.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: Saving service iscsi.datapool spec with placement smithi138=iscsi.a;count:1 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: pgmap v198: 132 pgs: 2 creating+activating, 18 creating+peering, 64 active+clean, 48 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2980940073' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1642360733' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:52.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T13:56:53.423 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:53 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:56:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:56:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T13:56:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T13:56:53.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T13:56:53.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:53.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:56:53.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T13:56:53.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T13:56:53.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:53.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T13:56:53.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T13:56:53.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:53 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.604+0000 7f76cdfc0700 1 -- 172.21.15.138:0/133603901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103470 msgr2=0x7f76c81038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.604+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/133603901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103470 0x7f76c81038d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f76b00099c0 tx=0x7f76b0009cd0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.604+0000 7f76cdfc0700 1 -- 172.21.15.138:0/133603901 shutdown_connections 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.604+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/133603901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c8102270 0x7f76c8102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.604+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/133603901 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76c8103e10 0x7f76c810a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.604+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/133603901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103470 0x7f76c81038d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.604+0000 7f76cdfc0700 1 -- 172.21.15.138:0/133603901 >> 172.21.15.138:0/133603901 conn(0x7f76c80fd850 msgr2=0x7f76c80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.605+0000 7f76cdfc0700 1 -- 172.21.15.138:0/133603901 shutdown_connections 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.605+0000 7f76cdfc0700 1 -- 172.21.15.138:0/133603901 wait complete. 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.605+0000 7f76cdfc0700 1 Processor -- start 2024-03-30T13:56:53.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.605+0000 7f76cdfc0700 1 -- start start 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c8102270 0x7f76c8075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76c8103470 0x7f76c8074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103e10 0x7f76c8074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76c810d580 con 0x7f76c8102270 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f76c810d400 con 0x7f76c8103e10 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f76c810d700 con 0x7f76c8103470 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103e10 0x7f76c8074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c8102270 0x7f76c8075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103e10 0x7f76c8074b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55864/0 (socket says 172.21.15.138:55864) 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 -- 172.21.15.138:0/2374182983 learned_addr learned my addr 172.21.15.138:0/2374182983 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c6ffd700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76c8103470 0x7f76c8074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 -- 172.21.15.138:0/2374182983 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76c8103470 msgr2=0x7f76c8074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76c8103470 0x7f76c8074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 -- 172.21.15.138:0/2374182983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c8102270 msgr2=0x7f76c8075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c8102270 0x7f76c8075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 -- 172.21.15.138:0/2374182983 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c8075420 con 0x7f76c8103e10 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c7fff700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103e10 0x7f76c8074b70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76c0007d70 tx=0x7f76c000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c4ff9700 1 -- 172.21.15.138:0/2374182983 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c0016020 con 0x7f76c8103e10 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c81a1c70 con 0x7f76c8103e10 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c4ff9700 1 -- 172.21.15.138:0/2374182983 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f76c000ac90 con 0x7f76c8103e10 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76c4ff9700 1 -- 172.21.15.138:0/2374182983 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c00115e0 con 0x7f76c8103e10 2024-03-30T13:56:53.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.606+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c81a2130 con 0x7f76c8103e10 2024-03-30T13:56:53.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.607+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c810dce0 con 0x7f76c8103e10 2024-03-30T13:56:53.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.608+0000 7f76c4ff9700 1 -- 172.21.15.138:0/2374182983 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f76c0019020 con 0x7f76c8103e10 2024-03-30T13:56:53.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.608+0000 7f76c4ff9700 1 --2- 172.21.15.138:0/2374182983 >> v2:172.21.15.102:6800/3154847229 conn(0x7f76b406db10 0x7f76b406ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:53.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.608+0000 7f76c4ff9700 1 -- 172.21.15.138:0/2374182983 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5942+0+0 (secure 0 0 0) 0x7f76c008d7b0 con 0x7f76c8103e10 2024-03-30T13:56:53.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.609+0000 7f76c77fe700 1 --2- 172.21.15.138:0/2374182983 >> v2:172.21.15.102:6800/3154847229 conn(0x7f76b406db10 0x7f76b406ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:53.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.609+0000 7f76c77fe700 1 --2- 172.21.15.138:0/2374182983 >> v2:172.21.15.102:6800/3154847229 conn(0x7f76b406db10 0x7f76b406ffd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f76c81032d0 tx=0x7f76b8007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:53.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.613+0000 7f76c4ff9700 1 -- 172.21.15.138:0/2374182983 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f76c0020020 con 0x7f76c8103e10 2024-03-30T13:56:53.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.825+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}) -- 0x7f76c802df10 con 0x7f76b406db10 2024-03-30T13:56:53.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.835+0000 7f76c4ff9700 1 -- 172.21.15.138:0/2374182983 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (secure 0 0 0) 0x7f76c802df10 con 0x7f76b406db10 2024-03-30T13:56:53.832 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled prometheus update... 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 >> v2:172.21.15.102:6800/3154847229 conn(0x7f76b406db10 msgr2=0x7f76b406ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/2374182983 >> v2:172.21.15.102:6800/3154847229 conn(0x7f76b406db10 0x7f76b406ffd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f76c81032d0 tx=0x7f76b8007400 comp rx=0 tx=0).stop 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103e10 msgr2=0x7f76c8074b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103e10 0x7f76c8074b70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76c0007d70 tx=0x7f76c000a340 comp rx=0 tx=0).stop 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 shutdown_connections 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/2374182983 >> v2:172.21.15.102:6800/3154847229 conn(0x7f76b406db10 0x7f76b406ffd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c8102270 0x7f76c8075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76c8103e10 0x7f76c8074b70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 --2- 172.21.15.138:0/2374182983 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76c8103470 0x7f76c8074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:53.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 >> 172.21.15.138:0/2374182983 conn(0x7f76c80fd850 msgr2=0x7f76c81067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:53.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 shutdown_connections 2024-03-30T13:56:53.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:53.837+0000 7f76cdfc0700 1 -- 172.21.15.138:0/2374182983 wait complete. 2024-03-30T13:56:54.281 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:54 smithi138 ceph-mon[33397]: Deploying daemon iscsi.iscsi.a on smithi138 2024-03-30T13:56:54.281 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:54 smithi138 ceph-mon[33397]: pgmap v201: 132 pgs: 4 creating+activating, 11 creating+peering, 99 active+clean, 18 unknown; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 1023 B/s wr, 18 op/s 2024-03-30T13:56:54.281 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:54 smithi138 ceph-mon[33397]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:56:54.281 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:54 smithi138 ceph-mon[33397]: Cluster is now healthy 2024-03-30T13:56:54.282 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:54 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:54.282 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:54 smithi138 ceph-mon[33397]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T13:56:54.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[37717]: Deploying daemon iscsi.iscsi.a on smithi138 2024-03-30T13:56:54.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[37717]: pgmap v201: 132 pgs: 4 creating+activating, 11 creating+peering, 99 active+clean, 18 unknown; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 1023 B/s wr, 18 op/s 2024-03-30T13:56:54.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[37717]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:56:54.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[37717]: Cluster is now healthy 2024-03-30T13:56:54.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:54.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[37717]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T13:56:54.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[29392]: Deploying daemon iscsi.iscsi.a on smithi138 2024-03-30T13:56:54.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[29392]: pgmap v201: 132 pgs: 4 creating+activating, 11 creating+peering, 99 active+clean, 18 unknown; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 1023 B/s wr, 18 op/s 2024-03-30T13:56:54.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[29392]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T13:56:54.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[29392]: Cluster is now healthy 2024-03-30T13:56:54.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:54.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:54 smithi102 ceph-mon[29392]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T13:56:54.584 DEBUG:teuthology.orchestra.run.smithi138:prometheus.a> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@prometheus.a.service 2024-03-30T13:56:54.586 INFO:tasks.cephadm:Adding node-exporter.a on smithi102 2024-03-30T13:56:54.586 INFO:tasks.cephadm:Adding node-exporter.b on smithi138 2024-03-30T13:56:54.586 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply node-exporter '2;smithi102=a;smithi138=b' 2024-03-30T13:56:55.195 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:55.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:55 smithi138 ceph-mon[33397]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:55.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:55 smithi138 ceph-mon[33397]: Saving service prometheus spec with placement smithi138=a;count:1 2024-03-30T13:56:55.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:55 smithi102 ceph-mon[29392]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:55.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:55 smithi102 ceph-mon[29392]: Saving service prometheus spec with placement smithi138=a;count:1 2024-03-30T13:56:55.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:55 smithi102 ceph-mon[37717]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:55.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:55 smithi102 ceph-mon[37717]: Saving service prometheus spec with placement smithi138=a;count:1 2024-03-30T13:56:55.936 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:55 smithi138 systemd[1]: Starting Ceph iscsi.iscsi.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:56:56.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:56 smithi138 ceph-mon[33397]: pgmap v203: 132 pgs: 4 creating+activating, 128 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 185 op/s 2024-03-30T13:56:56.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:56 smithi102 ceph-mon[29392]: pgmap v203: 132 pgs: 4 creating+activating, 128 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 185 op/s 2024-03-30T13:56:56.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:56 smithi102 ceph-mon[37717]: pgmap v203: 132 pgs: 4 creating+activating, 128 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 185 op/s 2024-03-30T13:56:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 -- 172.21.15.138:0/3026324319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708103eb0 msgr2=0x7f870810a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 --2- 172.21.15.138:0/3026324319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708103eb0 0x7f870810a760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f870000b0a0 tx=0x7f870000b3b0 comp rx=0 tx=0).stop 2024-03-30T13:56:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 -- 172.21.15.138:0/3026324319 shutdown_connections 2024-03-30T13:56:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 --2- 172.21.15.138:0/3026324319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8708100530 0x7f8708100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 --2- 172.21.15.138:0/3026324319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708103eb0 0x7f870810a760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 --2- 172.21.15.138:0/3026324319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8708103510 0x7f8708103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 -- 172.21.15.138:0/3026324319 >> 172.21.15.138:0/3026324319 conn(0x7f87080fbeb0 msgr2=0x7f87080fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 -- 172.21.15.138:0/3026324319 shutdown_connections 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.779+0000 7f870eab8700 1 -- 172.21.15.138:0/3026324319 wait complete. 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 Processor -- start 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 -- start start 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708100530 0x7f8708197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8708103510 0x7f8708198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8708103eb0 0x7f870819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f870810d590 con 0x7f8708103eb0 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f870810d410 con 0x7f8708100530 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870eab8700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f870810d710 con 0x7f8708103510 2024-03-30T13:56:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708100530 0x7f8708197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708100530 0x7f8708197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55882/0 (socket says 172.21.15.138:55882) 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 -- 172.21.15.138:0/2270739881 learned_addr learned my addr 172.21.15.138:0/2270739881 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f8707fff700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8708103510 0x7f8708198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870d055700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8708103eb0 0x7f870819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 -- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8708103510 msgr2=0x7f8708198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8708103510 0x7f8708198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 -- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8708103eb0 msgr2=0x7f870819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8708103eb0 0x7f870819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.780+0000 7f870c854700 1 -- 172.21.15.138:0/2270739881 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870819f780 con 0x7f8708100530 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.781+0000 7f870c854700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708100530 0x7f8708197b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00d980 tx=0x7f86fc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.781+0000 7f8705ffb700 1 -- 172.21.15.138:0/2270739881 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc017030 con 0x7f8708100530 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.781+0000 7f8705ffb700 1 -- 172.21.15.138:0/2270739881 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f86fc0044d0 con 0x7f8708100530 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.781+0000 7f8705ffb700 1 -- 172.21.15.138:0/2270739881 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc003980 con 0x7f8708100530 2024-03-30T13:56:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.781+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870819fa70 con 0x7f8708100530 2024-03-30T13:56:56.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.781+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87081a0000 con 0x7f8708100530 2024-03-30T13:56:56.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.782+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8708068f00 con 0x7f8708100530 2024-03-30T13:56:56.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.783+0000 7f8705ffb700 1 -- 172.21.15.138:0/2270739881 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f86fc01a070 con 0x7f8708100530 2024-03-30T13:56:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.783+0000 7f8705ffb700 1 --2- 172.21.15.138:0/2270739881 >> v2:172.21.15.102:6800/3154847229 conn(0x7f86f806db10 0x7f86f806ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.783+0000 7f8705ffb700 1 -- 172.21.15.138:0/2270739881 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5942+0+0 (secure 0 0 0) 0x7f86fc08d0e0 con 0x7f8708100530 2024-03-30T13:56:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.783+0000 7f8707fff700 1 --2- 172.21.15.138:0/2270739881 >> v2:172.21.15.102:6800/3154847229 conn(0x7f86f806db10 0x7f86f806ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.784+0000 7f8707fff700 1 --2- 172.21.15.138:0/2270739881 >> v2:172.21.15.102:6800/3154847229 conn(0x7f86f806db10 0x7f86f806ffd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f86f4004630 tx=0x7f86f400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.788+0000 7f8705ffb700 1 -- 172.21.15.138:0/2270739881 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f86fc05ad80 con 0x7f8708100530 2024-03-30T13:56:56.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:56.999+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi138=b", "target": ["mon-mgr", ""]}) -- 0x7f870802dea0 con 0x7f86f806db10 2024-03-30T13:56:57.002 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.006+0000 7f8705ffb700 1 -- 172.21.15.138:0/2270739881 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (secure 0 0 0) 0x7f870802dea0 con 0x7f86f806db10 2024-03-30T13:56:57.002 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled node-exporter update... 2024-03-30T13:56:57.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 >> v2:172.21.15.102:6800/3154847229 conn(0x7f86f806db10 msgr2=0x7f86f806ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:57.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 --2- 172.21.15.138:0/2270739881 >> v2:172.21.15.102:6800/3154847229 conn(0x7f86f806db10 0x7f86f806ffd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f86f4004630 tx=0x7f86f400b3c0 comp rx=0 tx=0).stop 2024-03-30T13:56:57.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708100530 msgr2=0x7f8708197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:57.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708100530 0x7f8708197b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00d980 tx=0x7f86fc00dc90 comp rx=0 tx=0).stop 2024-03-30T13:56:57.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 shutdown_connections 2024-03-30T13:56:57.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 --2- 172.21.15.138:0/2270739881 >> v2:172.21.15.102:6800/3154847229 conn(0x7f86f806db10 0x7f86f806ffd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:57.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8708103eb0 0x7f870819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:57.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8708100530 0x7f8708197b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:57.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 --2- 172.21.15.138:0/2270739881 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8708103510 0x7f8708198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:57.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 >> 172.21.15.138:0/2270739881 conn(0x7f87080fbeb0 msgr2=0x7f8708106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:57.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 shutdown_connections 2024-03-30T13:56:57.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:57.008+0000 7f870eab8700 1 -- 172.21.15.138:0/2270739881 wait complete. 2024-03-30T13:56:57.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:56 smithi138 podman[60878]: 2024-03-30 13:56:56.820773301 +0000 UTC m=+0.447811846 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T13:56:57.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:56 smithi138 podman[60878]: 2024-03-30 13:56:56.918345846 +0000 UTC m=+0.545384388 container create e61cb60d83a5f9c047d51ab3f987c138c3bdd3307bdbac736aebfea30260667d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-30T13:56:57.604 DEBUG:teuthology.orchestra.run.smithi102:node-exporter.a> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.a.service 2024-03-30T13:56:57.607 DEBUG:teuthology.orchestra.run.smithi138:node-exporter.b> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.b.service 2024-03-30T13:56:57.609 INFO:tasks.cephadm:Adding alertmanager.a on smithi102 2024-03-30T13:56:57.610 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply alertmanager '1;smithi102=a' 2024-03-30T13:56:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:57 smithi138 podman[60878]: 2024-03-30 13:56:57.758610345 +0000 UTC m=+1.385648890 container init e61cb60d83a5f9c047d51ab3f987c138c3bdd3307bdbac736aebfea30260667d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-30T13:56:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:57 smithi138 podman[60878]: 2024-03-30 13:56:57.763240409 +0000 UTC m=+1.390278951 container start e61cb60d83a5f9c047d51ab3f987c138c3bdd3307bdbac736aebfea30260667d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-30T13:56:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:57 smithi138 bash[60878]: e61cb60d83a5f9c047d51ab3f987c138c3bdd3307bdbac736aebfea30260667d 2024-03-30T13:56:57.935 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:57 smithi138 systemd[1]: Started Ceph iscsi.iscsi.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:56:57.953 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:56:58.274 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:58.274 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: Saving service node-exporter spec with placement smithi102=a;smithi138=b;count:2 2024-03-30T13:56:58.275 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.275 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.2 KiB/s wr, 188 op/s 2024-03-30T13:56:58.275 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.275 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.275 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.275 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.275 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug Started the configuration object watcher 2024-03-30T13:56:58.275 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug Checking for config object changes every 1s 2024-03-30T13:56:58.275 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug Processing osd blocklist entries for this node 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: Saving service node-exporter spec with placement smithi102=a;smithi138=b;count:2 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.2 KiB/s wr, 188 op/s 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi138=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: Saving service node-exporter spec with placement smithi102=a;smithi138=b;count:2 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.2 KiB/s wr, 188 op/s 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:58 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:56:58.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug Reading the configuration object to update local LIO configuration 2024-03-30T13:56:58.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug Configuration does not have an entry for this host(smithi138.front.sepia.ceph.com) - nothing to define to LIO 2024-03-30T13:56:58.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:56:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-30T13:56:59.340 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:59 smithi138 ceph-mon[33397]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T13:56:59.340 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:59 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:59.340 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:56:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/3050182293' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:59 smithi102 ceph-mon[29392]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:59 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:56:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/3050182293' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T13:56:59.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:59 smithi102 ceph-mon[37717]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T13:56:59.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:59 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:56:59.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:56:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/3050182293' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 -- 172.21.15.138:0/2296099288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 msgr2=0x7f7278102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 --2- 172.21.15.138:0/2296099288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f7278102c60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7260009990 tx=0x7f7260009ca0 comp rx=0 tx=0).stop 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 -- 172.21.15.138:0/2296099288 shutdown_connections 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 --2- 172.21.15.138:0/2296099288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7278101dd0 0x7f7278109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 --2- 172.21.15.138:0/2296099288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f7278102c60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 --2- 172.21.15.138:0/2296099288 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7278101410 0x7f7278101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 -- 172.21.15.138:0/2296099288 >> 172.21.15.138:0/2296099288 conn(0x7f72780fbee0 msgr2=0x7f72780fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:56:59.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 -- 172.21.15.138:0/2296099288 shutdown_connections 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.803+0000 7f727cf39700 1 -- 172.21.15.138:0/2296099288 wait complete. 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 Processor -- start 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 -- start start 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7278101410 0x7f72781a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7278101dd0 0x7f72781a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f72781a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f727810c530 con 0x7f7278101dd0 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f727810c3b0 con 0x7f7278102840 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f727cf39700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f727810c6b0 con 0x7f7278101410 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f7276d9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f72781a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f7276d9d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f72781a5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55974/0 (socket says 172.21.15.138:55974) 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f7276d9d700 1 -- 172.21.15.138:0/2076501933 learned_addr learned my addr 172.21.15.138:0/2076501933 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:56:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.804+0000 7f7275d9b700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7278101dd0 0x7f72781a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f727659c700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7278101410 0x7f72781a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f7276d9d700 1 -- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7278101410 msgr2=0x7f72781a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f7276d9d700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7278101410 0x7f72781a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f7276d9d700 1 -- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7278101dd0 msgr2=0x7f72781a33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f7276d9d700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7278101dd0 0x7f72781a33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f7276d9d700 1 -- 172.21.15.138:0/2076501933 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72781a5ef0 con 0x7f7278102840 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f7275d9b700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7278101dd0 0x7f72781a33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f7276d9d700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f72781a5930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f727000b970 tx=0x7f727000bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270019070 con 0x7f7278102840 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f7270014440 con 0x7f7278102840 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72781a61e0 con 0x7f7278102840 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727000ce60 con 0x7f7278102840 2024-03-30T13:56:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.805+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72781a67a0 con 0x7f7278102840 2024-03-30T13:56:59.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.806+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7278068f00 con 0x7f7278102840 2024-03-30T13:56:59.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.806+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 91740+0+0 (secure 0 0 0) 0x7f727001c070 con 0x7f7278102840 2024-03-30T13:56:59.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.807+0000 7f726f7fe700 1 --2- 172.21.15.138:0/2076501933 >> v2:172.21.15.102:6800/3154847229 conn(0x7f726406db10 0x7f726406ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:56:59.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.807+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5942+0+0 (secure 0 0 0) 0x7f727008cc90 con 0x7f7278102840 2024-03-30T13:56:59.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.807+0000 7f727659c700 1 --2- 172.21.15.138:0/2076501933 >> v2:172.21.15.102:6800/3154847229 conn(0x7f726406db10 0x7f726406ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:56:59.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.808+0000 7f727659c700 1 --2- 172.21.15.138:0/2076501933 >> v2:172.21.15.102:6800/3154847229 conn(0x7f726406db10 0x7f726406ffd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7260003730 tx=0x7f7260003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:56:59.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:56:59.811+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f727005a2e0 con 0x7f7278102840 2024-03-30T13:57:00.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.024+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mon.1 v2:172.21.15.138:3300/0 7 ==== mgrmap(e 16) ==== 91791+0+0 (secure 0 0 0) 0x7f7270056030 con 0x7f7278102840 2024-03-30T13:57:00.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.050+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}) -- 0x7f727802de80 con 0x7f726406db10 2024-03-30T13:57:00.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.057+0000 7f726f7fe700 1 -- 172.21.15.138:0/2076501933 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (secure 0 0 0) 0x7f727802de80 con 0x7f726406db10 2024-03-30T13:57:00.054 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled alertmanager update... 2024-03-30T13:57:00.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.059+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 >> v2:172.21.15.102:6800/3154847229 conn(0x7f726406db10 msgr2=0x7f726406ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:00.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 --2- 172.21.15.138:0/2076501933 >> v2:172.21.15.102:6800/3154847229 conn(0x7f726406db10 0x7f726406ffd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7260003730 tx=0x7f7260003680 comp rx=0 tx=0).stop 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 msgr2=0x7f72781a5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f72781a5930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f727000b970 tx=0x7f727000bd30 comp rx=0 tx=0).stop 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 shutdown_connections 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 --2- 172.21.15.138:0/2076501933 >> v2:172.21.15.102:6800/3154847229 conn(0x7f726406db10 0x7f726406ffd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7278102840 0x7f72781a5930 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7278101410 0x7f72781a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 --2- 172.21.15.138:0/2076501933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7278101dd0 0x7f72781a33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 >> 172.21.15.138:0/2076501933 conn(0x7f72780fbee0 msgr2=0x7f72780fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 shutdown_connections 2024-03-30T13:57:00.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:00.060+0000 7f727cf39700 1 -- 172.21.15.138:0/2076501933 wait complete. 2024-03-30T13:57:00.296 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:00 smithi138 ceph-mon[33397]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 142 op/s 2024-03-30T13:57:00.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:00 smithi102 ceph-mon[29392]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 142 op/s 2024-03-30T13:57:00.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:00 smithi102 ceph-mon[37717]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.7 KiB/s wr, 142 op/s 2024-03-30T13:57:01.235 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:01 smithi138 ceph-mon[33397]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T13:57:01.235 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:01 smithi138 ceph-mon[33397]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:57:01.235 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:01 smithi138 ceph-mon[33397]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-03-30T13:57:01.235 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:01 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:01.262 DEBUG:teuthology.orchestra.run.smithi102:alertmanager.a> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@alertmanager.a.service 2024-03-30T13:57:01.264 INFO:tasks.cephadm:Adding grafana.a on smithi138 2024-03-30T13:57:01.265 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph orch apply grafana '1;smithi138=a' 2024-03-30T13:57:01.289 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[37717]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T13:57:01.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[37717]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:57:01.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[37717]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-03-30T13:57:01.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:01.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[29392]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T13:57:01.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[29392]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:57:01.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[29392]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-03-30T13:57:01.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:01 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:02.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:02 smithi138 ceph-mon[33397]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.2 KiB/s wr, 126 op/s 2024-03-30T13:57:02.191 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:57:02.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:02 smithi102 ceph-mon[29392]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.2 KiB/s wr, 126 op/s 2024-03-30T13:57:02.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:02 smithi102 ceph-mon[37717]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.2 KiB/s wr, 126 op/s 2024-03-30T13:57:03.327 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.327 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.327 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:03 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:03 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:03 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:03.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/820908282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd4102d70 msgr2=0x7fcfd4109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:03.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/820908282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd4102d70 0x7fcfd4109620 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00b0d0 tx=0x7fcfcc00b3e0 comp rx=0 tx=0).stop 2024-03-30T13:57:03.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/820908282 shutdown_connections 2024-03-30T13:57:03.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/820908282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd4102d70 0x7fcfd4109620 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:03.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/820908282 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcfd41011d0 0x7fcfd41015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:03.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/820908282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 0x7fcfd4102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:03.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/820908282 >> 172.21.15.138:0/820908282 conn(0x7fcfd40fc790 msgr2=0x7fcfd40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:03.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/820908282 shutdown_connections 2024-03-30T13:57:03.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.775+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/820908282 wait complete. 2024-03-30T13:57:03.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 Processor -- start 2024-03-30T13:57:03.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 -- start start 2024-03-30T13:57:03.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd41011d0 0x7fcfd4196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:03.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 0x7fcfd4196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:03.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcfd4102d70 0x7fcfd419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd410c4d0 con 0x7fcfd41023d0 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd410c350 con 0x7fcfd41011d0 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd8fd0700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd410c650 con 0x7fcfd4102d70 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd1d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 0x7fcfd4196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd259c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd41011d0 0x7fcfd4196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd1d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 0x7fcfd4196f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.138:54824/0 (socket says 172.21.15.138:54824) 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd1d9b700 1 -- 172.21.15.138:0/2866260781 learned_addr learned my addr 172.21.15.138:0/2866260781 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.776+0000 7fcfd2d9d700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcfd4102d70 0x7fcfd419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd259c700 1 -- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcfd4102d70 msgr2=0x7fcfd419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd259c700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcfd4102d70 0x7fcfd419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd259c700 1 -- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 msgr2=0x7fcfd4196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:03.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd259c700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 0x7fcfd4196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd259c700 1 -- 172.21.15.138:0/2866260781 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd419e680 con 0x7fcfd41011d0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd1d9b700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 0x7fcfd4196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd2d9d700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcfd4102d70 0x7fcfd419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd259c700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd41011d0 0x7fcfd4196a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400e9c0 tx=0x7fcfc400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfc37fe700 1 -- 172.21.15.138:0/2866260781 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4004230 con 0x7fcfd41011d0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfc37fe700 1 -- 172.21.15.138:0/2866260781 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fcfc40043d0 con 0x7fcfd41011d0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd419e970 con 0x7fcfd41011d0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfc37fe700 1 -- 172.21.15.138:0/2866260781 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4003dc0 con 0x7fcfd41011d0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.777+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd419eeb0 con 0x7fcfd41011d0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.779+0000 7fcfc37fe700 1 -- 172.21.15.138:0/2866260781 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 91791+0+0 (secure 0 0 0) 0x7fcfc400c7c0 con 0x7fcfd41011d0 2024-03-30T13:57:03.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.779+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd4068f00 con 0x7fcfd41011d0 2024-03-30T13:57:03.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.779+0000 7fcfc37fe700 1 --2- 172.21.15.138:0/2866260781 >> v2:172.21.15.102:6800/3154847229 conn(0x7fcfbc06dba0 0x7fcfbc070060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:03.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.779+0000 7fcfc37fe700 1 -- 172.21.15.138:0/2866260781 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5942+0+0 (secure 0 0 0) 0x7fcfc408dc70 con 0x7fcfd41011d0 2024-03-30T13:57:03.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.779+0000 7fcfd1d9b700 1 --2- 172.21.15.138:0/2866260781 >> v2:172.21.15.102:6800/3154847229 conn(0x7fcfbc06dba0 0x7fcfbc070060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:03.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.780+0000 7fcfd1d9b700 1 --2- 172.21.15.138:0/2866260781 >> v2:172.21.15.102:6800/3154847229 conn(0x7fcfbc06dba0 0x7fcfbc070060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8011550 tx=0x7fcfc8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:03.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:03.784+0000 7fcfc37fe700 1 -- 172.21.15.138:0/2866260781 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcfc400b2c0 con 0x7fcfd41011d0 2024-03-30T13:57:03.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.002+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 --> v2:172.21.15.102:6800/3154847229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}) -- 0x7fcfd4062680 con 0x7fcfbc06dba0 2024-03-30T13:57:04.005 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled grafana update... 2024-03-30T13:57:04.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.008+0000 7fcfc37fe700 1 -- 172.21.15.138:0/2866260781 <== mgr.14152 v2:172.21.15.102:6800/3154847229 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fcfd4062680 con 0x7fcfbc06dba0 2024-03-30T13:57:04.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 >> v2:172.21.15.102:6800/3154847229 conn(0x7fcfbc06dba0 msgr2=0x7fcfbc070060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:04.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/2866260781 >> v2:172.21.15.102:6800/3154847229 conn(0x7fcfbc06dba0 0x7fcfbc070060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8011550 tx=0x7fcfc8009260 comp rx=0 tx=0).stop 2024-03-30T13:57:04.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd41011d0 msgr2=0x7fcfd4196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:04.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd41011d0 0x7fcfd4196a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400e9c0 tx=0x7fcfc400ecd0 comp rx=0 tx=0).stop 2024-03-30T13:57:04.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 shutdown_connections 2024-03-30T13:57:04.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/2866260781 >> v2:172.21.15.102:6800/3154847229 conn(0x7fcfbc06dba0 0x7fcfbc070060 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:04.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcfd4102d70 0x7fcfd419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:04.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfd41011d0 0x7fcfd4196a20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:04.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 --2- 172.21.15.138:0/2866260781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd41023d0 0x7fcfd4196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:04.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.010+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 >> 172.21.15.138:0/2866260781 conn(0x7fcfd40fc790 msgr2=0x7fcfd41056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:04.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.011+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 shutdown_connections 2024-03-30T13:57:04.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:04.011+0000 7fcfd8fd0700 1 -- 172.21.15.138:0/2866260781 wait complete. 2024-03-30T13:57:04.327 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:04 smithi138 ceph-mon[33397]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.4 KiB/s wr, 106 op/s 2024-03-30T13:57:04.327 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:04 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:57:04.327 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:04 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:57:04.327 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:04 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:04.328 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:04 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[29392]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.4 KiB/s wr, 106 op/s 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[37717]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.4 KiB/s wr, 106 op/s 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:57:04.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:04.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:04 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:04.572 DEBUG:teuthology.orchestra.run.smithi138:grafana.a> sudo journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@grafana.a.service 2024-03-30T13:57:04.574 INFO:tasks.cephadm:Setting up client nodes... 2024-03-30T13:57:04.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T13:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:05.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:05 smithi102 ceph-mon[29392]: Deploying daemon prometheus.a on smithi138 2024-03-30T13:57:05.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:05 smithi102 ceph-mon[29392]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:57:05.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:05 smithi102 ceph-mon[29392]: Saving service grafana spec with placement smithi138=a;count:1 2024-03-30T13:57:05.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:05 smithi102 ceph-mon[37717]: Deploying daemon prometheus.a on smithi138 2024-03-30T13:57:05.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:05 smithi102 ceph-mon[37717]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:57:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:05 smithi102 ceph-mon[37717]: Saving service grafana spec with placement smithi138=a;count:1 2024-03-30T13:57:05.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:05 smithi138 ceph-mon[33397]: Deploying daemon prometheus.a on smithi138 2024-03-30T13:57:05.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:05 smithi138 ceph-mon[33397]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi138=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T13:57:05.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:05 smithi138 ceph-mon[33397]: Saving service grafana spec with placement smithi138=a;count:1 2024-03-30T13:57:06.241 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:06 smithi102 ceph-mon[29392]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 826 B/s wr, 24 op/s 2024-03-30T13:57:06.242 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:06 smithi102 ceph-mon[37717]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 826 B/s wr, 24 op/s 2024-03-30T13:57:06.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:06 smithi138 ceph-mon[33397]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 826 B/s wr, 24 op/s 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 -- 172.21.15.102:0/796596493 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 msgr2=0x7ff1501026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 --2- 172.21.15.102:0/796596493 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 0x7ff1501026b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0098f0 tx=0x7ff14c009c00 comp rx=0 tx=0).stop 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 -- 172.21.15.102:0/796596493 shutdown_connections 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 --2- 172.21.15.102:0/796596493 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff150103e30 0x7ff15010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 --2- 172.21.15.102:0/796596493 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 0x7ff1501026b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 --2- 172.21.15.102:0/796596493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103490 0x7ff1501038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 -- 172.21.15.102:0/796596493 >> 172.21.15.102:0/796596493 conn(0x7ff1500fd850 msgr2=0x7ff1500ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 -- 172.21.15.102:0/796596493 shutdown_connections 2024-03-30T13:57:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 -- 172.21.15.102:0/796596493 wait complete. 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 Processor -- start 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.687+0000 7ff158341700 1 -- start start 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 0x7ff150197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff150103490 0x7ff1501980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103e30 0x7ff15019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d5c0 con 0x7ff150103e30 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d440 con 0x7ff150103490 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff15010d740 con 0x7ff150102290 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103e30 0x7ff15019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1560dd700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 0x7ff150197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103e30 0x7ff15019f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41486/0 (socket says 172.21.15.102:41486) 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 -- 172.21.15.102:0/422292590 learned_addr learned my addr 172.21.15.102:0/422292590 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1558dc700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff150103490 0x7ff1501980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 -- 172.21.15.102:0/422292590 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 msgr2=0x7ff150197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 0x7ff150197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 -- 172.21.15.102:0/422292590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff150103490 msgr2=0x7ff1501980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:06.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff150103490 0x7ff1501980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 -- 172.21.15.102:0/422292590 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15019f7f0 con 0x7ff150103e30 2024-03-30T13:57:06.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1568de700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103e30 0x7ff15019f150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff14800ef60 tx=0x7ff14800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1477fe700 1 -- 172.21.15.102:0/422292590 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1480092e0 con 0x7ff150103e30 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1477fe700 1 -- 172.21.15.102:0/422292590 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7ff148004bb0 con 0x7ff150103e30 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15019fa80 con 0x7ff150103e30 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff1477fe700 1 -- 172.21.15.102:0/422292590 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148004020 con 0x7ff150103e30 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.688+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15019ff70 con 0x7ff150103e30 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.689+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff150050750 con 0x7ff150103e30 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.690+0000 7ff1477fe700 1 -- 172.21.15.102:0/422292590 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 91791+0+0 (secure 0 0 0) 0x7ff14801b030 con 0x7ff150103e30 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.690+0000 7ff1477fe700 1 --2- 172.21.15.102:0/422292590 >> v2:172.21.15.102:6800/3154847229 conn(0x7ff13c06dc40 0x7ff13c070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.690+0000 7ff1477fe700 1 -- 172.21.15.102:0/422292590 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5942+0+0 (secure 0 0 0) 0x7ff14808d4f0 con 0x7ff150103e30 2024-03-30T13:57:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.690+0000 7ff1560dd700 1 --2- 172.21.15.102:0/422292590 >> v2:172.21.15.102:6800/3154847229 conn(0x7ff13c06dc40 0x7ff13c070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.691+0000 7ff1560dd700 1 --2- 172.21.15.102:0/422292590 >> v2:172.21.15.102:6800/3154847229 conn(0x7ff13c06dc40 0x7ff13c070100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff14c009810 tx=0x7ff14c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.695+0000 7ff1477fe700 1 -- 172.21.15.102:0/422292590 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff148022020 con 0x7ff150103e30 2024-03-30T13:57:06.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.970+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7ff15002db30 con 0x7ff150103e30 2024-03-30T13:57:06.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.972+0000 7ff1477fe700 1 -- 172.21.15.102:0/422292590 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) ==== 170+0+59 (secure 0 0 0) 0x7ff148010040 con 0x7ff150103e30 2024-03-30T13:57:06.968 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-03-30T13:57:06.968 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQAyGghm7h/pORAAZhWKqBIU3UfKMsgK/28wVw== 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 >> v2:172.21.15.102:6800/3154847229 conn(0x7ff13c06dc40 msgr2=0x7ff13c070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 --2- 172.21.15.102:0/422292590 >> v2:172.21.15.102:6800/3154847229 conn(0x7ff13c06dc40 0x7ff13c070100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff14c009810 tx=0x7ff14c019040 comp rx=0 tx=0).stop 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103e30 msgr2=0x7ff15019f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103e30 0x7ff15019f150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff14800ef60 tx=0x7ff14800ef90 comp rx=0 tx=0).stop 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 shutdown_connections 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 --2- 172.21.15.102:0/422292590 >> v2:172.21.15.102:6800/3154847229 conn(0x7ff13c06dc40 0x7ff13c070100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff150103e30 0x7ff15019f150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff150102290 0x7ff150197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 --2- 172.21.15.102:0/422292590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff150103490 0x7ff1501980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.974+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 >> 172.21.15.102:0/422292590 conn(0x7ff1500fd850 msgr2=0x7ff150106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:06.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.975+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 shutdown_connections 2024-03-30T13:57:06.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:06.975+0000 7ff158341700 1 -- 172.21.15.102:0/422292590 wait complete. 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/422292590' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/422292590' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/422292590' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/422292590' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:57:07.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-30T13:57:07.299 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-03-30T13:57:07.299 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:07 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-30T13:57:07.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/422292590' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:07.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/422292590' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:57:07.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:07 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-30T13:57:07.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:07 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-03-30T13:57:07.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:07 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-30T13:57:07.619 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:57:07.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-30T13:57:07.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T13:57:07.664 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T13:57:07.970 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T13:57:08.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:08 smithi138 ceph-mon[33397]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 597 B/s wr, 19 op/s 2024-03-30T13:57:08.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-30T13:57:08.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-03-30T13:57:08.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:08 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-30T13:57:08.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:08 smithi138 ceph-mon[33397]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T13:57:08.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:57:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:57:08.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[29392]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 597 B/s wr, 19 op/s 2024-03-30T13:57:08.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-30T13:57:08.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-03-30T13:57:08.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-30T13:57:08.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[29392]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T13:57:08.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[37717]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 597 B/s wr, 19 op/s 2024-03-30T13:57:08.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-30T13:57:08.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-03-30T13:57:08.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-30T13:57:08.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:08 smithi102 ceph-mon[37717]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T13:57:09.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:09 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:09.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:09 smithi138 ceph-mon[33397]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T13:57:09.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:09 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:09.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:09 smithi102 ceph-mon[29392]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T13:57:09.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:09 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:09.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:09 smithi102 ceph-mon[37717]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T13:57:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:10 smithi138 ceph-mon[33397]: pgmap v212: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-30T13:57:10.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:10 smithi102 ceph-mon[29392]: pgmap v212: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-30T13:57:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:10 smithi102 ceph-mon[37717]: pgmap v212: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-30T13:57:12.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:12 smithi138 ceph-mon[33397]: pgmap v213: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:57:12.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:12 smithi102 ceph-mon[29392]: pgmap v213: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:57:12.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:12 smithi102 ceph-mon[37717]: pgmap v213: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.713+0000 7feede73b700 1 -- 172.21.15.138:0/529071161 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103620 msgr2=0x7feed8103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.713+0000 7feede73b700 1 --2- 172.21.15.138:0/529071161 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103620 0x7feed8103a80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7feecc0099c0 tx=0x7feecc009cd0 comp rx=0 tx=0).stop 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 -- 172.21.15.138:0/529071161 shutdown_connections 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 --2- 172.21.15.138:0/529071161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed81005a0 0x7feed81009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 --2- 172.21.15.138:0/529071161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feed8103fc0 0x7feed810a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 --2- 172.21.15.138:0/529071161 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103620 0x7feed8103a80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 -- 172.21.15.138:0/529071161 >> 172.21.15.138:0/529071161 conn(0x7feed80fbee0 msgr2=0x7feed80fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 -- 172.21.15.138:0/529071161 shutdown_connections 2024-03-30T13:57:12.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 -- 172.21.15.138:0/529071161 wait complete. 2024-03-30T13:57:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 Processor -- start 2024-03-30T13:57:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.714+0000 7feede73b700 1 -- start start 2024-03-30T13:57:12.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feede73b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feed81005a0 0x7feed8197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feede73b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed8103620 0x7feed81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feede73b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103fc0 0x7feed819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feede73b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feed810d6a0 con 0x7feed8103620 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feede73b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feed810d520 con 0x7feed8103fc0 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feede73b700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7feed810d820 con 0x7feed81005a0 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103fc0 0x7feed819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103fc0 0x7feed819f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:56388/0 (socket says 172.21.15.138:56388) 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 -- 172.21.15.138:0/3779242129 learned_addr learned my addr 172.21.15.138:0/3779242129 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feed77fe700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed8103620 0x7feed81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feed7fff700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feed81005a0 0x7feed8197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 -- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feed81005a0 msgr2=0x7feed8197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feed81005a0 0x7feed8197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 -- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed8103620 msgr2=0x7feed81981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:12.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed8103620 0x7feed81981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 -- 172.21.15.138:0/3779242129 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed819f8c0 con 0x7feed8103fc0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feedccd8700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103fc0 0x7feed819f1f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feed000b970 tx=0x7feed000bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.715+0000 7feed57fa700 1 -- 172.21.15.138:0/3779242129 <== mon.1 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed000d860 con 0x7feed8103fc0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.716+0000 7feed57fa700 1 -- 172.21.15.138:0/3779242129 <== mon.1 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7feed000e430 con 0x7feed8103fc0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.716+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed819fbb0 con 0x7feed8103fc0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.716+0000 7feed57fa700 1 -- 172.21.15.138:0/3779242129 <== mon.1 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0017660 con 0x7feed8103fc0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.716+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed81a00f0 con 0x7feed8103fc0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.717+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed8050750 con 0x7feed8103fc0 2024-03-30T13:57:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.717+0000 7feed57fa700 1 -- 172.21.15.138:0/3779242129 <== mon.1 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 91791+0+0 (secure 0 0 0) 0x7feed000da00 con 0x7feed8103fc0 2024-03-30T13:57:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.717+0000 7feed57fa700 1 --2- 172.21.15.138:0/3779242129 >> v2:172.21.15.102:6800/3154847229 conn(0x7feec006dc40 0x7feec0070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.718+0000 7feed57fa700 1 -- 172.21.15.138:0/3779242129 <== mon.1 v2:172.21.15.138:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6001+0+0 (secure 0 0 0) 0x7feed008d5b0 con 0x7feed8103fc0 2024-03-30T13:57:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.718+0000 7feed7fff700 1 --2- 172.21.15.138:0/3779242129 >> v2:172.21.15.102:6800/3154847229 conn(0x7feec006dc40 0x7feec0070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.718+0000 7feed7fff700 1 --2- 172.21.15.138:0/3779242129 >> v2:172.21.15.102:6800/3154847229 conn(0x7feec006dc40 0x7feec0070100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feed8199150 tx=0x7feec8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:12.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.722+0000 7feed57fa700 1 -- 172.21.15.138:0/3779242129 <== mon.1 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feed0013030 con 0x7feed8103fc0 2024-03-30T13:57:12.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:12.999+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7feed802df80 con 0x7feed8103fc0 2024-03-30T13:57:13.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.004+0000 7feed57fa700 1 -- 172.21.15.138:0/3779242129 <== mon.1 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (secure 0 0 0) 0x7feed005b110 con 0x7feed8103fc0 2024-03-30T13:57:13.001 INFO:teuthology.orchestra.run.smithi138.stdout:[client.1] 2024-03-30T13:57:13.001 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQA5Gghmwr4YABAA3WdA3vL40A6GI3GTWoyzgA== 2024-03-30T13:57:13.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 >> v2:172.21.15.102:6800/3154847229 conn(0x7feec006dc40 msgr2=0x7feec0070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:13.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 --2- 172.21.15.138:0/3779242129 >> v2:172.21.15.102:6800/3154847229 conn(0x7feec006dc40 0x7feec0070100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feed8199150 tx=0x7feec8009190 comp rx=0 tx=0).stop 2024-03-30T13:57:13.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103fc0 msgr2=0x7feed819f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:13.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103fc0 0x7feed819f1f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feed000b970 tx=0x7feed000bc80 comp rx=0 tx=0).stop 2024-03-30T13:57:13.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 shutdown_connections 2024-03-30T13:57:13.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 --2- 172.21.15.138:0/3779242129 >> v2:172.21.15.102:6800/3154847229 conn(0x7feec006dc40 0x7feec0070100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:13.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feed8103fc0 0x7feed819f1f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:13.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feed81005a0 0x7feed8197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:13.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 --2- 172.21.15.138:0/3779242129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed8103620 0x7feed81981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:13.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.006+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 >> 172.21.15.138:0/3779242129 conn(0x7feed80fbee0 msgr2=0x7feed80fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:13.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.007+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 shutdown_connections 2024-03-30T13:57:13.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-03-30T13:57:13.007+0000 7feede73b700 1 -- 172.21.15.138:0/3779242129 wait complete. 2024-03-30T13:57:13.287 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.138:0/3779242129' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:13.287 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:13.287 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:57:13.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.138:0/3779242129' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:13.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:13.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:57:13.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.138:0/3779242129' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:13.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T13:57:13.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T13:57:14.419 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:14 smithi138 ceph-mon[33397]: pgmap v214: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T13:57:14.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:14 smithi102 ceph-mon[29392]: pgmap v214: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T13:57:14.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:14 smithi102 ceph-mon[37717]: pgmap v214: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T13:57:15.126 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 systemd[1]: Starting Ceph prometheus.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:57:15.470 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-03-30T13:57:15.470 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-30T13:57:15.470 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-30T13:57:15.514 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T13:57:15.514 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-30T13:57:15.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-30T13:57:15.763 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:15.948 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 podman[62498]: 2024-03-30 13:57:15.697502942 +0000 UTC m=+0.109266326 container create ce20971fd7ba01703e62de2658a28a88cf58d76b017b589059168e00fd8ebac1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:57:15.948 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 podman[62498]: 2024-03-30 13:57:15.606706252 +0000 UTC m=+0.018469662 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-30T13:57:15.948 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 podman[62498]: 2024-03-30 13:57:15.924595148 +0000 UTC m=+0.336358530 container init ce20971fd7ba01703e62de2658a28a88cf58d76b017b589059168e00fd8ebac1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:57:15.948 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 podman[62498]: 2024-03-30 13:57:15.927069777 +0000 UTC m=+0.338833161 container start ce20971fd7ba01703e62de2658a28a88cf58d76b017b589059168e00fd8ebac1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:57:16.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:16 smithi138 ceph-mon[33397]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-30T13:57:16.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:16 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T13:57:16.435 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.952Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.952Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.952Z caller=main.go:570 level=info host_details="(Linux 5.14.0-434.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024 x86_64 smithi138 (none))" 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.952Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.952Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.953Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.953Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.954Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.954Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.957Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.957Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.647µs 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.957Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.957Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.957Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.137µs wal_replay_duration=303.953µs wbl_replay_duration=123ns total_replay_duration=339.963µs 2024-03-30T13:57:16.436 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.958Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-30T13:57:16.437 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.958Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-30T13:57:16.437 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.958Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-30T13:57:16.437 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.976Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.999038ms db_storage=791ns remote_storage=1.256µs web_handler=376ns query_engine=621ns scrape=383.084µs scrape_sd=67.772µs notify=945ns notify_sd=1.913µs rules=17.380339ms tracing=5.501µs 2024-03-30T13:57:16.437 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.976Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-30T13:57:16.437 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:57:15.976Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-30T13:57:16.437 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:16 smithi138 bash[62498]: ce20971fd7ba01703e62de2658a28a88cf58d76b017b589059168e00fd8ebac1 2024-03-30T13:57:16.437 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:57:16 smithi138 systemd[1]: Started Ceph prometheus.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:57:16.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[29392]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[37717]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:16.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:16 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T13:57:17.188 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ignoring --setuser ceph since I am not root 2024-03-30T13:57:17.189 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ignoring --setgroup ceph since I am not root 2024-03-30T13:57:17.435 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: ignoring --setuser ceph since I am not root 2024-03-30T13:57:17.435 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: ignoring --setgroup ceph since I am not root 2024-03-30T13:57:17.435 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:17 smithi138 ceph-mgr[35270]: -- 172.21.15.138:0/2215964936 <== mon.1 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562b1bb6e5a0 con 0x562b1bb42800 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 -- 172.21.15.102:0/647491116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e81064c0 msgr2=0x7f24e810e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 --2- 172.21.15.102:0/647491116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e81064c0 0x7f24e810e090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f24e000b0d0 tx=0x7f24e000b3e0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 -- 172.21.15.102:0/647491116 shutdown_connections 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 --2- 172.21.15.102:0/647491116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e8101ba0 0x7f24e8101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 --2- 172.21.15.102:0/647491116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e81064c0 0x7f24e810e090 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 --2- 172.21.15.102:0/647491116 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f24e8102550 0x7f24e8105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 -- 172.21.15.102:0/647491116 >> 172.21.15.102:0/647491116 conn(0x7f24e8077740 msgr2=0x7f24e80feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 -- 172.21.15.102:0/647491116 shutdown_connections 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.574+0000 7f24ee81b700 1 -- 172.21.15.102:0/647491116 wait complete. 2024-03-30T13:57:17.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 Processor -- start 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 -- start start 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e8101ba0 0x7f24e8196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f24e8102550 0x7f24e8197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e81064c0 0x7f24e819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24e8110bc0 con 0x7f24e81064c0 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f24e8110a40 con 0x7f24e8102550 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ee81b700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f24e8110d40 con 0x7f24e8101ba0 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24e7fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e8101ba0 0x7f24e8196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24e7fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e8101ba0 0x7f24e8196fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:58684/0 (socket says 172.21.15.102:58684) 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24e7fff700 1 -- 172.21.15.102:0/3367246446 learned_addr learned my addr 172.21.15.102:0/3367246446 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24ecdb8700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e81064c0 0x7f24e819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.575+0000 7f24e77fe700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f24e8102550 0x7f24e8197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e7fff700 1 -- 172.21.15.102:0/3367246446 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f24e8102550 msgr2=0x7f24e8197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:17.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e7fff700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f24e8102550 0x7f24e8197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e7fff700 1 -- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e81064c0 msgr2=0x7f24e819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e7fff700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e81064c0 0x7f24e819b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e7fff700 1 -- 172.21.15.102:0/3367246446 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e819bfd0 con 0x7f24e8101ba0 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e7fff700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e8101ba0 0x7f24e8196fc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00eb30 tx=0x7f24dc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e57fa700 1 -- 172.21.15.102:0/3367246446 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00cc90 con 0x7f24e8101ba0 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e57fa700 1 -- 172.21.15.102:0/3367246446 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f24dc0044d0 con 0x7f24e8101ba0 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e819c2c0 con 0x7f24e8101ba0 2024-03-30T13:57:17.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24e57fa700 1 -- 172.21.15.102:0/3367246446 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc010430 con 0x7f24e8101ba0 2024-03-30T13:57:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.576+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e81a3b40 con 0x7f24e8101ba0 2024-03-30T13:57:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.577+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e8102950 con 0x7f24e8101ba0 2024-03-30T13:57:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.578+0000 7f24e57fa700 1 -- 172.21.15.102:0/3367246446 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 17) ==== 91805+0+0 (secure 0 0 0) 0x7f24dc003ec0 con 0x7f24e8101ba0 2024-03-30T13:57:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.578+0000 7f24e57fa700 1 --2- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 0x7f24d00700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.578+0000 7f24e77fe700 1 -- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 msgr2=0x7f24d00700b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3154847229 2024-03-30T13:57:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.578+0000 7f24e57fa700 1 -- 172.21.15.102:0/3367246446 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 6001+0+0 (secure 0 0 0) 0x7f24dc08dce0 con 0x7f24e8101ba0 2024-03-30T13:57:17.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.578+0000 7f24e77fe700 1 --2- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 0x7f24d00700b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:57:17.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.583+0000 7f24e57fa700 1 -- 172.21.15.102:0/3367246446 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f24dc05b7c0 con 0x7f24e8101ba0 2024-03-30T13:57:17.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.779+0000 7f24e77fe700 1 -- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 msgr2=0x7f24d00700b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3154847229 2024-03-30T13:57:17.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.779+0000 7f24e77fe700 1 --2- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 0x7f24d00700b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T13:57:17.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.840+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f24e8068f00 con 0x7f24e8101ba0 2024-03-30T13:57:17.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:17.670+0000 7ff4d3acd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:57:17.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:17.766+0000 7ff4d3acd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:57:17.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.845+0000 7f24e57fa700 1 -- 172.21.15.102:0/3367246446 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+175405 (secure 0 0 0) 0x7f24dc05b0c0 con 0x7f24e8101ba0 2024-03-30T13:57:17.843 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 msgr2=0x7f24d00700b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 --2- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 0x7f24d00700b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e8101ba0 msgr2=0x7f24e8196fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e8101ba0 0x7f24e8196fc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00eb30 tx=0x7f24dc00ee40 comp rx=0 tx=0).stop 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 shutdown_connections 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 --2- 172.21.15.102:0/3367246446 >> v2:172.21.15.102:6800/3154847229 conn(0x7f24d006dbf0 0x7f24d00700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e81064c0 0x7f24e819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e8101ba0 0x7f24e8196fc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 --2- 172.21.15.102:0/3367246446 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f24e8102550 0x7f24e8197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 >> 172.21.15.102:0/3367246446 conn(0x7f24e8077740 msgr2=0x7f24e8103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 shutdown_connections 2024-03-30T13:57:17.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:17.847+0000 7f24ee81b700 1 -- 172.21.15.102:0/3367246446 wait complete. 2024-03-30T13:57:17.934 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:17.659+0000 7f9a4264c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T13:57:17.935 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:17.753+0000 7f9a4264c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[37717]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[37717]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3367246446' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[29392]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[29392]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T13:57:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3367246446' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T13:57:18.387 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' 2024-03-30T13:57:18.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-mon[33397]: from='mgr.14152 172.21.15.102:0/3099834509' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T13:57:18.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-mon[33397]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T13:57:18.388 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3367246446' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T13:57:18.609 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":3154847229}]},"active_addr":"172.21.15.102:6800/3154847229","active_change":"2024-03-30T13:52:06.991238+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2379855428}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3355988508}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2496292408}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":4208971386}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3324784940}]}]} 2024-03-30T13:57:18.613 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-30T13:57:18.613 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T13:57:18.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T13:57:18.666 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:18.391+0000 7f9a4264c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:57:18.666 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:18.488+0000 7f9a4264c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:57:18.666 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:18.636+0000 7f9a4264c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:57:18.688 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:18.435+0000 7ff4d3acd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T13:57:18.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:18.529+0000 7ff4d3acd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T13:57:18.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:18.675+0000 7ff4d3acd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T13:57:18.866 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:18.935 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:18.737+0000 7f9a4264c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:57:18.936 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:18 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:18.826+0000 7f9a4264c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:57:19.024 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:18.774+0000 7ff4d3acd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T13:57:19.024 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:18 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:18.893+0000 7ff4d3acd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T13:57:19.283 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:19 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:19.171+0000 7ff4d3acd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:57:19.283 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:19 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:19.285+0000 7ff4d3acd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:57:19.398 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:19.101+0000 7f9a4264c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T13:57:19.398 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:19.216+0000 7f9a4264c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T13:57:19.398 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:19.304+0000 7f9a4264c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:57:19.563 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:19 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:19.374+0000 7ff4d3acd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T13:57:19.563 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:19 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:19.468+0000 7ff4d3acd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:57:19.685 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:19.401+0000 7f9a4264c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T13:57:19.685 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:19.584+0000 7f9a4264c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:57:19.939 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:19 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:19.648+0000 7ff4d3acd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T13:57:20.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 -- 172.21.15.102:0/1028049093 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec108360 msgr2=0x7f69ec108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/1028049093 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec108360 0x7f69ec108740 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f69dc009990 tx=0x7f69dc009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 -- 172.21.15.102:0/1028049093 shutdown_connections 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/1028049093 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f69ec102cb0 0x7f69ec10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/1028049093 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec108360 0x7f69ec108740 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/1028049093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69ec1022f0 0x7f69ec102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 -- 172.21.15.102:0/1028049093 >> 172.21.15.102:0/1028049093 conn(0x7f69ec079310 msgr2=0x7f69ec079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.556+0000 7f69f1aaf700 1 -- 172.21.15.102:0/1028049093 shutdown_connections 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.557+0000 7f69f1aaf700 1 -- 172.21.15.102:0/1028049093 wait complete. 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.557+0000 7f69f1aaf700 1 Processor -- start 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.557+0000 7f69f1aaf700 1 -- start start 2024-03-30T13:57:20.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec1022f0 0x7f69ec198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f69ec102cb0 0x7f69ec1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69ec108360 0x7f69ec19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec111c40 con 0x7f69ec108360 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec111ac0 con 0x7f69ec102cb0 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f69ec111dc0 con 0x7f69ec1022f0 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec1022f0 0x7f69ec198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec1022f0 0x7f69ec198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:43736/0 (socket says 172.21.15.102:43736) 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 -- 172.21.15.102:0/2895084513 learned_addr learned my addr 172.21.15.102:0/2895084513 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eaffd700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f69ec102cb0 0x7f69ec1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69ebfff700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69ec108360 0x7f69ec19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 -- 172.21.15.102:0/2895084513 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f69ec102cb0 msgr2=0x7f69ec1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f69ec102cb0 0x7f69ec1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 -- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69ec108360 msgr2=0x7f69ec19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:20.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69ec108360 0x7f69ec19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 -- 172.21.15.102:0/2895084513 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec19d0a0 con 0x7f69ec1022f0 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eaffd700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f69ec102cb0 0x7f69ec1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69eb7fe700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec1022f0 0x7f69ec198090 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f69dc0057f0 tx=0x7f69dc00bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69e8ff9700 1 -- 172.21.15.102:0/2895084513 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc026070 con 0x7f69ec1022f0 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69e8ff9700 1 -- 172.21.15.102:0/2895084513 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f69dc021410 con 0x7f69ec1022f0 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ec19d330 con 0x7f69ec1022f0 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69e8ff9700 1 -- 172.21.15.102:0/2895084513 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc017650 con 0x7f69ec1022f0 2024-03-30T13:57:20.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.558+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ec1a4c10 con 0x7f69ec1022f0 2024-03-30T13:57:20.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.560+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ec068f00 con 0x7f69ec1022f0 2024-03-30T13:57:20.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.560+0000 7f69e8ff9700 1 -- 172.21.15.102:0/2895084513 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 17) ==== 91805+0+0 (secure 0 0 0) 0x7f69dc029070 con 0x7f69ec1022f0 2024-03-30T13:57:20.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.560+0000 7f69e8ff9700 1 --2- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 0x7f69d40700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:20.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.560+0000 7f69eaffd700 1 -- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 msgr2=0x7f69d40700b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3154847229 2024-03-30T13:57:20.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.560+0000 7f69e8ff9700 1 -- 172.21.15.102:0/2895084513 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 6001+0+0 (secure 0 0 0) 0x7f69dc08ea00 con 0x7f69ec1022f0 2024-03-30T13:57:20.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.560+0000 7f69eaffd700 1 --2- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 0x7f69d40700b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:57:20.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.565+0000 7f69e8ff9700 1 -- 172.21.15.102:0/2895084513 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f69dc013070 con 0x7f69ec1022f0 2024-03-30T13:57:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.761+0000 7f69eaffd700 1 -- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 msgr2=0x7f69d40700b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3154847229 2024-03-30T13:57:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.761+0000 7f69eaffd700 1 --2- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 0x7f69d40700b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T13:57:20.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.766+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69ec1a4de0 con 0x7f69ec1022f0 2024-03-30T13:57:20.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.767+0000 7f69e8ff9700 1 -- 172.21.15.102:0/2895084513 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+20717 (secure 0 0 0) 0x7f69dc05c4e0 con 0x7f69ec1022f0 2024-03-30T13:57:20.765 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:57:20.765 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":71,"fsid":"69dd7ec8-ee9c-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:51:13.147429+0000","modified":"2024-03-30T13:57:08.979399+0000","last_up_change":"2024-03-30T13:56:27.720377+0000","last_in_change":"2024-03-30T13:56:09.983286+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:54:25.314887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T13:56:42.708873+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T13:56:43.357941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T13:56:45.426089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T13:56:47.377808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T13:56:49.440996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d9e04b0b-a4df-4d4f-87f8-59e29edd2e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6801","nonce":3100565996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3100565996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3100565996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6803","nonce":3100565996}]},"public_addr":"172.21.15.102:6801/3100565996","cluster_addr":"172.21.15.102:6802/3100565996","heartbeat_back_addr":"172.21.15.102:6804/3100565996","heartbeat_front_addr":"172.21.15.102:6803/3100565996","state":["exists","up"]},{"osd":1,"uuid":"046093f3-4186-496a-a5b2-9e8d4b86a2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":3593563159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3593563159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3593563159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6807","nonce":3593563159}]},"public_addr":"172.21.15.102:6805/3593563159","cluster_addr":"172.21.15.102:6806/3593563159","heartbeat_back_addr":"172.21.15.102:6808/3593563159","heartbeat_front_addr":"172.21.15.102:6807/3593563159","state":["exists","up"]},{"osd":2,"uuid":"4779686a-2976-4755-9116-8835f28fa5a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":203063794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":203063794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":203063794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6811","nonce":203063794}]},"public_addr":"172.21.15.102:6809/203063794","cluster_addr":"172.21.15.102:6810/203063794","heartbeat_back_addr":"172.21.15.102:6812/203063794","heartbeat_front_addr":"172.21.15.102:6811/203063794","state":["exists","up"]},{"osd":3,"uuid":"2cd657e0-4c72-45db-a26b-d86c06d2996b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":1080330048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1080330048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1080330048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6815","nonce":1080330048}]},"public_addr":"172.21.15.102:6813/1080330048","cluster_addr":"172.21.15.102:6814/1080330048","heartbeat_back_addr":"172.21.15.102:6816/1080330048","heartbeat_front_addr":"172.21.15.102:6815/1080330048","state":["exists","up"]},{"osd":4,"uuid":"a69453af-c657-49a7-b759-bebd7c7fefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":1718735591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":1718735591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":1718735591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":1718735591}]},"public_addr":"172.21.15.138:6800/1718735591","cluster_addr":"172.21.15.138:6801/1718735591","heartbeat_back_addr":"172.21.15.138:6803/1718735591","heartbeat_front_addr":"172.21.15.138:6802/1718735591","state":["exists","up"]},{"osd":5,"uuid":"2b723775-3bc5-4c26-8c7b-570cfe9a270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":1568298628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":1568298628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":1568298628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":1568298628}]},"public_addr":"172.21.15.138:6804/1568298628","cluster_addr":"172.21.15.138:6805/1568298628","heartbeat_back_addr":"172.21.15.138:6807/1568298628","heartbeat_front_addr":"172.21.15.138:6806/1568298628","state":["exists","up"]},{"osd":6,"uuid":"9cdfc0c0-2f91-44d9-9c0a-401e328da8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3214961690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":3214961690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":3214961690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3214961690}]},"public_addr":"172.21.15.138:6808/3214961690","cluster_addr":"172.21.15.138:6809/3214961690","heartbeat_back_addr":"172.21.15.138:6811/3214961690","heartbeat_front_addr":"172.21.15.138:6810/3214961690","state":["exists","up"]},{"osd":7,"uuid":"67ed0818-2486-420e-97ff-0264fac6a48b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3765635388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":3765635388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":3765635388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3765635388}]},"public_addr":"172.21.15.138:6812/3765635388","cluster_addr":"172.21.15.138:6813/3765635388","heartbeat_back_addr":"172.21.15.138:6815/3765635388","heartbeat_front_addr":"172.21.15.138:6814/3765635388","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:53:30.706660+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:53:55.797625+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:54:21.798206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:54:48.334156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:55:11.795826+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:55:36.167708+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:56:01.269021+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:56:25.786109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1457123288":"2024-03-31T13:52:06.991127+0000","172.21.15.102:0/1446807484":"2024-03-31T13:52:06.991127+0000","172.21.15.102:0/10326931":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/2083482926":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/1947582370":"2024-03-31T13:51:40.671168+0000","172.21.15.102:6800/2520146638":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/335601191":"2024-03-31T13:52:06.991127+0000","172.21.15.102:6800/3737793096":"2024-03-31T13:52:06.991127+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 msgr2=0x7f69d40700b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 0x7f69d40700b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec1022f0 msgr2=0x7f69ec198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec1022f0 0x7f69ec198090 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f69dc0057f0 tx=0x7f69dc00bef0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 shutdown_connections 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/2895084513 >> v2:172.21.15.102:6800/3154847229 conn(0x7f69d406dbf0 0x7f69d40700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69ec108360 0x7f69ec19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f69ec1022f0 0x7f69ec198090 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 --2- 172.21.15.102:0/2895084513 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f69ec102cb0 0x7f69ec1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 >> 172.21.15.102:0/2895084513 conn(0x7f69ec079310 msgr2=0x7f69ec105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 shutdown_connections 2024-03-30T13:57:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:20.771+0000 7f69f1aaf700 1 -- 172.21.15.102:0/2895084513 wait complete. 2024-03-30T13:57:20.816 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:20 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:20.636+0000 7ff4d3acd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:57:20.816 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:20 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:20.771+0000 7ff4d3acd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:57:20.817 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:20.543+0000 7f9a4264c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T13:57:20.817 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:20.674+0000 7f9a4264c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T13:57:21.095 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2895084513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:57:21.096 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:21.009+0000 7f9a4264c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:57:21.121 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2895084513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:57:21.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2895084513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:57:21.412 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:21 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:21.124+0000 7ff4d3acd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T13:57:21.413 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:21 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:21.217+0000 7ff4d3acd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:57:21.434 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:21.098+0000 7f9a4264c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T13:57:21.435 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:21.297+0000 7f9a4264c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:57:21.566 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T13:57:21.567 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T13:57:21.688 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:21 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:21.415+0000 7ff4d3acd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:57:21.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:21 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:21.608+0000 7ff4d3acd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:57:21.763 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:21.481+0000 7f9a4264c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T13:57:21.812 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:22.026 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:21.766+0000 7f9a4264c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:57:22.176 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:21 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:21.916+0000 7ff4d3acd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T13:57:22.427 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:22 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:22.179+0000 7ff4d3acd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:57:22.434 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:22 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:22.030+0000 7f9a4264c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T13:57:22.434 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:22 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:22.267+0000 7f9a4264c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:57:22.688 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:22 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:22.430+0000 7ff4d3acd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T13:57:23.435 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:23 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:23.063+0000 7f9a4264c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:57:23.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/4222697426 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc06a890 msgr2=0x7fa1dc10f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/4222697426 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc06a890 0x7fa1dc10f0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400b0d0 tx=0x7fa1d400b3e0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/4222697426 shutdown_connections 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/4222697426 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc06a890 0x7fa1dc10f0d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/4222697426 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1dc108290 0x7fa1dc108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/4222697426 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1dc069ef0 0x7fa1dc06a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/4222697426 >> 172.21.15.102:0/4222697426 conn(0x7fa1dc078070 msgr2=0x7fa1dc078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/4222697426 shutdown_connections 2024-03-30T13:57:23.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.533+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/4222697426 wait complete. 2024-03-30T13:57:23.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 Processor -- start 2024-03-30T13:57:23.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 -- start start 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc069ef0 0x7fa1dc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1dc06a890 0x7fa1dc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1dc108290 0x7fa1dc19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc111c10 con 0x7fa1dc108290 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e087c700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc069ef0 0x7fa1dc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc111a90 con 0x7fa1dc06a890 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e2ae0700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa1dc111d90 con 0x7fa1dc069ef0 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e087c700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc069ef0 0x7fa1dc197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:43752/0 (socket says 172.21.15.102:43752) 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e087c700 1 -- 172.21.15.102:0/93103241 learned_addr learned my addr 172.21.15.102:0/93103241 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1dbfff700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1dc06a890 0x7fa1dc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:23.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.534+0000 7fa1e107d700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1dc108290 0x7fa1dc19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:23.534 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:23.284+0000 7ff4d3acd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e087c700 1 -- 172.21.15.102:0/93103241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1dc06a890 msgr2=0x7fa1dc198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e087c700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1dc06a890 0x7fa1dc198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e087c700 1 -- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1dc108290 msgr2=0x7fa1dc19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e087c700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1dc108290 0x7fa1dc19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e087c700 1 -- 172.21.15.102:0/93103241 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc19cfe0 con 0x7fa1dc069ef0 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1dbfff700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1dc06a890 0x7fa1dc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e107d700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1dc108290 0x7fa1dc19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e087c700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc069ef0 0x7fa1dc197fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000e9a0 tx=0x7fa1d000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:23.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1d9ffb700 1 -- 172.21.15.102:0/93103241 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0020070 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1dc19d2d0 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1d9ffb700 1 -- 172.21.15.102:0/93103241 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fa1d0013d00 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.535+0000 7fa1d9ffb700 1 -- 172.21.15.102:0/93103241 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d001b7e0 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.536+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1dc101140 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.536+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1dc068f00 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.537+0000 7fa1d9ffb700 1 -- 172.21.15.102:0/93103241 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 17) ==== 91805+0+0 (secure 0 0 0) 0x7fa1d000c480 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.537+0000 7fa1d9ffb700 1 --2- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 0x7fa1cc0788c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.538+0000 7fa1dbfff700 1 -- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 msgr2=0x7fa1cc0788c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3154847229 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.538+0000 7fa1d9ffb700 1 -- 172.21.15.102:0/93103241 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 6001+0+0 (secure 0 0 0) 0x7fa1d008d710 con 0x7fa1dc069ef0 2024-03-30T13:57:23.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.538+0000 7fa1dbfff700 1 --2- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 0x7fa1cc0788c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T13:57:23.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.542+0000 7fa1d9ffb700 1 -- 172.21.15.102:0/93103241 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa1d005b300 con 0x7fa1dc069ef0 2024-03-30T13:57:23.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.739+0000 7fa1dbfff700 1 -- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 msgr2=0x7fa1cc0788c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3154847229 2024-03-30T13:57:23.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.739+0000 7fa1dbfff700 1 --2- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 0x7fa1cc0788c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T13:57:23.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.747+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1dc101400 con 0x7fa1dc069ef0 2024-03-30T13:57:23.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.749+0000 7fa1d9ffb700 1 -- 172.21.15.102:0/93103241 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+20717 (secure 0 0 0) 0x7fa1d005ac00 con 0x7fa1dc069ef0 2024-03-30T13:57:23.746 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:57:23.746 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":71,"fsid":"69dd7ec8-ee9c-11ee-b647-cb9ed24678a4","created":"2024-03-30T13:51:13.147429+0000","modified":"2024-03-30T13:57:08.979399+0000","last_up_change":"2024-03-30T13:56:27.720377+0000","last_in_change":"2024-03-30T13:56:09.983286+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T13:54:25.314887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T13:56:42.708873+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T13:56:43.357941+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T13:56:45.426089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T13:56:47.377808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T13:56:49.440996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d9e04b0b-a4df-4d4f-87f8-59e29edd2e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6801","nonce":3100565996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":3100565996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":3100565996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6803","nonce":3100565996}]},"public_addr":"172.21.15.102:6801/3100565996","cluster_addr":"172.21.15.102:6802/3100565996","heartbeat_back_addr":"172.21.15.102:6804/3100565996","heartbeat_front_addr":"172.21.15.102:6803/3100565996","state":["exists","up"]},{"osd":1,"uuid":"046093f3-4186-496a-a5b2-9e8d4b86a2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6805","nonce":3593563159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":3593563159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3593563159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6807","nonce":3593563159}]},"public_addr":"172.21.15.102:6805/3593563159","cluster_addr":"172.21.15.102:6806/3593563159","heartbeat_back_addr":"172.21.15.102:6808/3593563159","heartbeat_front_addr":"172.21.15.102:6807/3593563159","state":["exists","up"]},{"osd":2,"uuid":"4779686a-2976-4755-9116-8835f28fa5a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":203063794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":203063794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":203063794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6811","nonce":203063794}]},"public_addr":"172.21.15.102:6809/203063794","cluster_addr":"172.21.15.102:6810/203063794","heartbeat_back_addr":"172.21.15.102:6812/203063794","heartbeat_front_addr":"172.21.15.102:6811/203063794","state":["exists","up"]},{"osd":3,"uuid":"2cd657e0-4c72-45db-a26b-d86c06d2996b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":1080330048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1080330048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1080330048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6815","nonce":1080330048}]},"public_addr":"172.21.15.102:6813/1080330048","cluster_addr":"172.21.15.102:6814/1080330048","heartbeat_back_addr":"172.21.15.102:6816/1080330048","heartbeat_front_addr":"172.21.15.102:6815/1080330048","state":["exists","up"]},{"osd":4,"uuid":"a69453af-c657-49a7-b759-bebd7c7fefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":1718735591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":1718735591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":1718735591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":1718735591}]},"public_addr":"172.21.15.138:6800/1718735591","cluster_addr":"172.21.15.138:6801/1718735591","heartbeat_back_addr":"172.21.15.138:6803/1718735591","heartbeat_front_addr":"172.21.15.138:6802/1718735591","state":["exists","up"]},{"osd":5,"uuid":"2b723775-3bc5-4c26-8c7b-570cfe9a270d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":1568298628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":1568298628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":1568298628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":1568298628}]},"public_addr":"172.21.15.138:6804/1568298628","cluster_addr":"172.21.15.138:6805/1568298628","heartbeat_back_addr":"172.21.15.138:6807/1568298628","heartbeat_front_addr":"172.21.15.138:6806/1568298628","state":["exists","up"]},{"osd":6,"uuid":"9cdfc0c0-2f91-44d9-9c0a-401e328da8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3214961690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":3214961690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":3214961690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3214961690}]},"public_addr":"172.21.15.138:6808/3214961690","cluster_addr":"172.21.15.138:6809/3214961690","heartbeat_back_addr":"172.21.15.138:6811/3214961690","heartbeat_front_addr":"172.21.15.138:6810/3214961690","state":["exists","up"]},{"osd":7,"uuid":"67ed0818-2486-420e-97ff-0264fac6a48b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3765635388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":3765635388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":3765635388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3765635388}]},"public_addr":"172.21.15.138:6812/3765635388","cluster_addr":"172.21.15.138:6813/3765635388","heartbeat_back_addr":"172.21.15.138:6815/3765635388","heartbeat_front_addr":"172.21.15.138:6814/3765635388","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:53:30.706660+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:53:55.797625+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:54:21.798206+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:54:48.334156+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:55:11.795826+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:55:36.167708+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:56:01.269021+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:56:25.786109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/1457123288":"2024-03-31T13:52:06.991127+0000","172.21.15.102:0/1446807484":"2024-03-31T13:52:06.991127+0000","172.21.15.102:0/10326931":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/2083482926":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/1947582370":"2024-03-31T13:51:40.671168+0000","172.21.15.102:6800/2520146638":"2024-03-31T13:51:40.671168+0000","172.21.15.102:0/335601191":"2024-03-31T13:52:06.991127+0000","172.21.15.102:6800/3737793096":"2024-03-31T13:52:06.991127+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:57:23.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 msgr2=0x7fa1cc0788c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T13:57:23.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 0x7fa1cc0788c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc069ef0 msgr2=0x7fa1dc197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc069ef0 0x7fa1dc197fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000e9a0 tx=0x7fa1d000ecb0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 shutdown_connections 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/93103241 >> v2:172.21.15.102:6800/3154847229 conn(0x7fa1cc076400 0x7fa1cc0788c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1dc108290 0x7fa1dc19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1dc069ef0 0x7fa1dc197fd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 --2- 172.21.15.102:0/93103241 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa1dc06a890 0x7fa1dc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 >> 172.21.15.102:0/93103241 conn(0x7fa1dc078070 msgr2=0x7fa1dc105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 shutdown_connections 2024-03-30T13:57:23.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:23.751+0000 7fa1e2ae0700 1 -- 172.21.15.102:0/93103241 wait complete. 2024-03-30T13:57:23.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/93103241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:57:23.935 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:23 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:23.636+0000 7f9a4264c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:57:23.935 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:23 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:23.735+0000 7f9a4264c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:57:24.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/93103241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:57:24.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/93103241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T13:57:24.189 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:23.874+0000 7ff4d3acd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T13:57:24.189 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:23.974+0000 7ff4d3acd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T13:57:24.313 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:23 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:23.952+0000 7f9a4264c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:57:24.313 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:23 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: [30/Mar/2024:13:57:23] ENGINE Bus STARTING 2024-03-30T13:57:24.313 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: CherryPy Checker: 2024-03-30T13:57:24.313 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: The Application mounted at '' has an empty config. 2024-03-30T13:57:24.313 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: 2024-03-30T13:57:24.313 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: [30/Mar/2024:13:57:24] ENGINE Serving on http://:::9283 2024-03-30T13:57:24.314 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x[35266]: [30/Mar/2024:13:57:24] ENGINE Bus STARTED 2024-03-30T13:57:24.573 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:24.195+0000 7ff4d3acd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T13:57:24.573 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:57:24] ENGINE Bus STARTING 2024-03-30T13:57:24.573 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: CherryPy Checker: 2024-03-30T13:57:24.573 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: The Application mounted at '' has an empty config. 2024-03-30T13:57:24.574 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T13:57:24.574 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:57:24] ENGINE Serving on http://:::9283 2024-03-30T13:57:24.574 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:57:24] ENGINE Bus STARTED 2024-03-30T13:57:24.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T13:57:24.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T13:57:24.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T13:57:24.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T13:57:24.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T13:57:24.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T13:57:24.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T13:57:24.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T13:57:24.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: Standby manager daemon x restarted 2024-03-30T13:57:24.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: Standby manager daemon x started 2024-03-30T13:57:24.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: Active manager daemon y restarted 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: Activating manager daemon y 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: mgrmap e18: y(active, starting, since 0.00766188s), standbys: x 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:57:24.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: Manager daemon y is now available 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:57:24.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: Standby manager daemon x restarted 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: Standby manager daemon x started 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: Active manager daemon y restarted 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: Activating manager daemon y 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T13:57:24.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: mgrmap e18: y(active, starting, since 0.00766188s), standbys: x 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:57:24.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: Manager daemon y is now available 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:57:24.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:57:25.084 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:25.094 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:25.098 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:25.113 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:25.119 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:25.134 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: Standby manager daemon x restarted 2024-03-30T13:57:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: Standby manager daemon x started 2024-03-30T13:57:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T13:57:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.? 172.21.15.138:0/2421325701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: Active manager daemon y restarted 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: Activating manager daemon y 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: mgrmap e18: y(active, starting, since 0.00766188s), standbys: x 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T13:57:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: Manager daemon y is now available 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:25.187 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:57:25.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T13:57:25.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:57:25.188 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T13:57:25.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:57:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:57:26.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[29392]: [30/Mar/2024:13:57:24] ENGINE Bus STARTING 2024-03-30T13:57:26.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[29392]: [30/Mar/2024:13:57:24] ENGINE Serving on https://172.21.15.102:7150 2024-03-30T13:57:26.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[29392]: [30/Mar/2024:13:57:24] ENGINE Serving on http://172.21.15.102:8765 2024-03-30T13:57:26.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[29392]: [30/Mar/2024:13:57:24] ENGINE Bus STARTED 2024-03-30T13:57:26.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[29392]: mgrmap e19: y(active, since 1.01085s), standbys: x 2024-03-30T13:57:26.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[37717]: [30/Mar/2024:13:57:24] ENGINE Bus STARTING 2024-03-30T13:57:26.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[37717]: [30/Mar/2024:13:57:24] ENGINE Serving on https://172.21.15.102:7150 2024-03-30T13:57:26.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[37717]: [30/Mar/2024:13:57:24] ENGINE Serving on http://172.21.15.102:8765 2024-03-30T13:57:26.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[37717]: [30/Mar/2024:13:57:24] ENGINE Bus STARTED 2024-03-30T13:57:26.066 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:25 smithi102 ceph-mon[37717]: mgrmap e19: y(active, since 1.01085s), standbys: x 2024-03-30T13:57:26.170 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:25 smithi138 ceph-mon[33397]: [30/Mar/2024:13:57:24] ENGINE Bus STARTING 2024-03-30T13:57:26.170 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:25 smithi138 ceph-mon[33397]: [30/Mar/2024:13:57:24] ENGINE Serving on https://172.21.15.102:7150 2024-03-30T13:57:26.170 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:25 smithi138 ceph-mon[33397]: [30/Mar/2024:13:57:24] ENGINE Serving on http://172.21.15.102:8765 2024-03-30T13:57:26.170 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:25 smithi138 ceph-mon[33397]: [30/Mar/2024:13:57:24] ENGINE Bus STARTED 2024-03-30T13:57:26.170 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:25 smithi138 ceph-mon[33397]: mgrmap e19: y(active, since 1.01085s), standbys: x 2024-03-30T13:57:26.790 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:26.815 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:57:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:27.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:27.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:26 smithi138 ceph-mon[33397]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:27.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:27.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:26 smithi102 ceph-mon[29392]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:27.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:27.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:26 smithi102 ceph-mon[37717]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:28.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:27 smithi138 ceph-mon[33397]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:28.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:27 smithi138 ceph-mon[33397]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-30T13:57:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:27 smithi102 ceph-mon[37717]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:27 smithi102 ceph-mon[37717]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-30T13:57:28.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:27 smithi102 ceph-mon[29392]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:28.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:27 smithi102 ceph-mon[29392]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-30T13:57:30.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[29392]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:30.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[29392]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-30T13:57:30.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:30.080 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:30.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[37717]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:30.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[37717]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-30T13:57:30.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:30.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:29 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:30.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:29 smithi138 ceph-mon[33397]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:57:30.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:29 smithi138 ceph-mon[33397]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-30T13:57:30.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:29 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:30.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:29 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi138 to 3760M 2024-03-30T13:57:31.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi138 to 3760M 2024-03-30T13:57:31.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:31.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi138 to 3760M 2024-03-30T13:57:31.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:32.175 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:31 smithi102 ceph-mon[29392]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-30T13:57:32.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:31 smithi102 ceph-mon[37717]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-30T13:57:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:31 smithi138 ceph-mon[33397]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-30T13:57:33.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 -- 172.21.15.102:0/1480762579 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c107270 msgr2=0x7fed7c107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:33.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 --2- 172.21.15.102:0/1480762579 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c107270 0x7fed7c107650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fed7400b0a0 tx=0x7fed7400b3b0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 -- 172.21.15.102:0/1480762579 shutdown_connections 2024-03-30T13:57:33.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 --2- 172.21.15.102:0/1480762579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed7c101de0 0x7fed7c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 --2- 172.21.15.102:0/1480762579 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c107270 0x7fed7c107650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 --2- 172.21.15.102:0/1480762579 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed7c101420 0x7fed7c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 -- 172.21.15.102:0/1480762579 >> 172.21.15.102:0/1480762579 conn(0x7fed7c078000 msgr2=0x7fed7c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 -- 172.21.15.102:0/1480762579 shutdown_connections 2024-03-30T13:57:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.353+0000 7fed8180e700 1 -- 172.21.15.102:0/1480762579 wait complete. 2024-03-30T13:57:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 Processor -- start 2024-03-30T13:57:33.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 -- start start 2024-03-30T13:57:33.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed7c101420 0x7fed7c102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:33.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c101de0 0x7fed7c103310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:33.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed7c107270 0x7fed7c103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:33.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c110af0 con 0x7fed7c101420 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c110970 con 0x7fed7c107270 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed8180e700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fed7c110c70 con 0x7fed7c101de0 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed7a7fc700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c101de0 0x7fed7c103310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed7a7fc700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c101de0 0x7fed7c103310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:47128/0 (socket says 172.21.15.102:47128) 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed7a7fc700 1 -- 172.21.15.102:0/2337391427 learned_addr learned my addr 172.21.15.102:0/2337391427 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.354+0000 7fed7affd700 1 --2- 172.21.15.102:0/2337391427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed7c101420 0x7fed7c102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7b7fe700 1 --2- 172.21.15.102:0/2337391427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed7c107270 0x7fed7c103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7a7fc700 1 -- 172.21.15.102:0/2337391427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed7c107270 msgr2=0x7fed7c103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7a7fc700 1 --2- 172.21.15.102:0/2337391427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed7c107270 0x7fed7c103850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7a7fc700 1 -- 172.21.15.102:0/2337391427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed7c101420 msgr2=0x7fed7c102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7a7fc700 1 --2- 172.21.15.102:0/2337391427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed7c101420 0x7fed7c102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7a7fc700 1 -- 172.21.15.102:0/2337391427 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7c1a6600 con 0x7fed7c101de0 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7affd700 1 --2- 172.21.15.102:0/2337391427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed7c101420 0x7fed7c102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:33.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed7a7fc700 1 --2- 172.21.15.102:0/2337391427 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c101de0 0x7fed7c103310 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00e9c0 tx=0x7fed6c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00cbd0 con 0x7fed7c101de0 2024-03-30T13:57:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fed6c00cd70 con 0x7fed7c101de0 2024-03-30T13:57:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c005630 con 0x7fed7c101de0 2024-03-30T13:57:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7c1a68f0 con 0x7fed7c101de0 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.355+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed7c1a6de0 con 0x7fed7c101de0 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.357+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fed7c100940 con 0x7fed7c101de0 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.357+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fed6c004750 con 0x7fed7c101de0 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.357+0000 7fed8080c700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.102:6800/622906077 conn(0x7fed6806dc00 0x7fed680700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7fed6c08ddc0 con 0x7fed7c101de0 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed7affd700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.102:6800/622906077 conn(0x7fed6806dc00 0x7fed680700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed8080c700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.138:6804/1568298628 conn(0x7fed680774a0 0x7fed68079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 --> v2:172.21.15.138:6804/1568298628 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed6c019e70 con 0x7fed680774a0 2024-03-30T13:57:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fed6c0904c0 con 0x7fed7c101de0 2024-03-30T13:57:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed7affd700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.102:6800/622906077 conn(0x7fed6806dc00 0x7fed680700c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed74009e60 tx=0x7fed7400b700 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed7b7fe700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.138:6804/1568298628 conn(0x7fed680774a0 0x7fed68079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.358+0000 7fed7b7fe700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.138:6804/1568298628 conn(0x7fed680774a0 0x7fed68079940 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.359+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== osd.5 v2:172.21.15.138:6804/1568298628 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed6c019e70 con 0x7fed680774a0 2024-03-30T13:57:33.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.375+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 --> v2:172.21.15.138:6804/1568298628 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fed7c102870 con 0x7fed680774a0 2024-03-30T13:57:33.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.376+0000 7fed8080c700 1 -- 172.21.15.102:0/2337391427 <== osd.5 v2:172.21.15.138:6804/1568298628 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fed7c102870 con 0x7fed680774a0 2024-03-30T13:57:33.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.376+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 >> v2:172.21.15.138:6804/1568298628 conn(0x7fed680774a0 msgr2=0x7fed68079940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.376+0000 7fed8180e700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.138:6804/1568298628 conn(0x7fed680774a0 0x7fed68079940 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 >> v2:172.21.15.102:6800/622906077 conn(0x7fed6806dc00 msgr2=0x7fed680700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 --2- 172.21.15.102:0/2337391427 >> v2:172.21.15.102:6800/622906077 conn(0x7fed6806dc00 0x7fed680700c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed74009e60 tx=0x7fed7400b700 comp rx=0 tx=0).stop 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c101de0 msgr2=0x7fed7c103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 --2- 172.21.15.102:0/2337391427 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed7c101de0 0x7fed7c103310 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00e9c0 tx=0x7fed6c00ecd0 comp rx=0 tx=0).stop 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed7b7fe700 1 -- 172.21.15.102:0/2337391427 reap_dead start 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 shutdown_connections 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 >> 172.21.15.102:0/2337391427 conn(0x7fed7c078000 msgr2=0x7fed7c0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 shutdown_connections 2024-03-30T13:57:33.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:33.377+0000 7fed8180e700 1 -- 172.21.15.102:0/2337391427 wait complete. 2024-03-30T13:57:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 -- 172.21.15.102:0/1533424821 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 msgr2=0x7f9ecc101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 --2- 172.21.15.102:0/1533424821 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 0x7f9ecc101020 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009990 tx=0x7f9ec0009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 -- 172.21.15.102:0/1533424821 shutdown_connections 2024-03-30T13:57:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 --2- 172.21.15.102:0/1533424821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ecc1130d0 0x7f9ecc1154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 --2- 172.21.15.102:0/1533424821 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 0x7f9ecc101020 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 --2- 172.21.15.102:0/1533424821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ecc101560 0x7f9ecc112b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 -- 172.21.15.102:0/1533424821 >> 172.21.15.102:0/1533424821 conn(0x7f9ecc076490 msgr2=0x7f9ecc0768a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 -- 172.21.15.102:0/1533424821 shutdown_connections 2024-03-30T13:57:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 -- 172.21.15.102:0/1533424821 wait complete. 2024-03-30T13:57:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 Processor -- start 2024-03-30T13:57:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.010+0000 7f9ed2378700 1 -- start start 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed2378700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 0x7f9ecc1a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed2378700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ecc101560 0x7f9ecc1a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed2378700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ecc1130d0 0x7f9ecc19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed2378700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc118330 con 0x7f9ecc101560 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed2378700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1181b0 con 0x7f9ecc1130d0 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed2378700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9ecc1184b0 con 0x7f9ecc100c40 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 0x7f9ecc1a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 0x7f9ecc1a4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:47150/0 (socket says 172.21.15.102:47150) 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 -- 172.21.15.102:0/1283949585 learned_addr learned my addr 172.21.15.102:0/1283949585 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecb7fe700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ecc101560 0x7f9ecc1a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed0915700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ecc1130d0 0x7f9ecc19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 -- 172.21.15.102:0/1283949585 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ecc1130d0 msgr2=0x7f9ecc19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ecc1130d0 0x7f9ecc19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 -- 172.21.15.102:0/1283949585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ecc101560 msgr2=0x7f9ecc1a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ecc101560 0x7f9ecc1a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 -- 172.21.15.102:0/1283949585 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ecc19ed30 con 0x7f9ecc100c40 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ed0915700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ecc1130d0 0x7f9ecc19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecb7fe700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ecc101560 0x7f9ecc1a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.011+0000 7f9ecbfff700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 0x7f9ecc1a4410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0000c00 tx=0x7f9ec00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.012+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0027070 con 0x7f9ecc100c40 2024-03-30T13:57:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.012+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f9ec0022470 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.012+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ecc19efc0 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.012+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0017650 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.012+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ecc1ab2e0 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.013+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ecc111560 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.013+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f9ec002a070 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.013+0000 7f9ec97fa700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6800/622906077 conn(0x7f9eb406dc00 0x7f9eb40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.013+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f9ec008eab0 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.014+0000 7f9ecb7fe700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6800/622906077 conn(0x7f9eb406dc00 0x7f9eb40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.014+0000 7f9ec97fa700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6805/3593563159 conn(0x7f9eb40774a0 0x7f9eb4079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.014+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 --> v2:172.21.15.102:6805/3593563159 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ec000b0b0 con 0x7f9eb40774a0 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.014+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f9ec00932c0 con 0x7f9ecc100c40 2024-03-30T13:57:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.014+0000 7f9ed0915700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6805/3593563159 conn(0x7f9eb40774a0 0x7f9eb4079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.014+0000 7f9ecb7fe700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6800/622906077 conn(0x7f9eb406dc00 0x7f9eb40700c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc112c20 tx=0x7f9ebc00a300 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.014+0000 7f9ed0915700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6805/3593563159 conn(0x7f9eb40774a0 0x7f9eb4079940 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.015+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== osd.1 v2:172.21.15.102:6805/3593563159 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ec000b0b0 con 0x7f9eb40774a0 2024-03-30T13:57:34.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.031+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 --> v2:172.21.15.102:6805/3593563159 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9ecc06a4b0 con 0x7f9eb40774a0 2024-03-30T13:57:34.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.031+0000 7f9ec97fa700 1 -- 172.21.15.102:0/1283949585 <== osd.1 v2:172.21.15.102:6805/3593563159 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9ecc06a4b0 con 0x7f9eb40774a0 2024-03-30T13:57:34.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.031+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6805/3593563159 conn(0x7f9eb40774a0 msgr2=0x7f9eb4079940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.031+0000 7f9ed2378700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6805/3593563159 conn(0x7f9eb40774a0 0x7f9eb4079940 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6800/622906077 conn(0x7f9eb406dc00 msgr2=0x7f9eb40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 --2- 172.21.15.102:0/1283949585 >> v2:172.21.15.102:6800/622906077 conn(0x7f9eb406dc00 0x7f9eb40700c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc112c20 tx=0x7f9ebc00a300 comp rx=0 tx=0).stop 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 msgr2=0x7f9ecc1a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 --2- 172.21.15.102:0/1283949585 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ecc100c40 0x7f9ecc1a4410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0000c00 tx=0x7f9ec00047b0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed0915700 1 -- 172.21.15.102:0/1283949585 reap_dead start 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 shutdown_connections 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 >> 172.21.15.102:0/1283949585 conn(0x7f9ecc076490 msgr2=0x7f9ecc110b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 shutdown_connections 2024-03-30T13:57:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.032+0000 7f9ed2378700 1 -- 172.21.15.102:0/1283949585 wait complete. 2024-03-30T13:57:34.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:33 smithi102 ceph-mon[37717]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T13:57:34.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:33 smithi102 ceph-mon[29392]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T13:57:34.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:33 smithi138 ceph-mon[33397]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T13:57:34.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 -- 172.21.15.102:0/96135695 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740101c90 msgr2=0x7f5740102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 --2- 172.21.15.102:0/96135695 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740101c90 0x7f5740102110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5734009990 tx=0x7f5734009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 -- 172.21.15.102:0/96135695 shutdown_connections 2024-03-30T13:57:34.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 --2- 172.21.15.102:0/96135695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106180 0x7f5740106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 --2- 172.21.15.102:0/96135695 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5740102650 0x7f574010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 --2- 172.21.15.102:0/96135695 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740101c90 0x7f5740102110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 -- 172.21.15.102:0/96135695 >> 172.21.15.102:0/96135695 conn(0x7f5740076480 msgr2=0x7f5740076890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:34.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 -- 172.21.15.102:0/96135695 shutdown_connections 2024-03-30T13:57:34.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.593+0000 7f5744d67700 1 -- 172.21.15.102:0/96135695 wait complete. 2024-03-30T13:57:34.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 Processor -- start 2024-03-30T13:57:34.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 -- start start 2024-03-30T13:57:34.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5740101c90 0x7f5740198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740102650 0x7f57401986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106180 0x7f574019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5740110bd0 con 0x7f5740106180 2024-03-30T13:57:34.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5740110a50 con 0x7f5740101c90 2024-03-30T13:57:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.594+0000 7f5744d67700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5740110d50 con 0x7f5740102650 2024-03-30T13:57:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106180 0x7f574019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573dd9b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740102650 0x7f57401986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106180 0x7f574019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44538/0 (socket says 172.21.15.102:44538) 2024-03-30T13:57:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573dd9b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740102650 0x7f57401986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:47170/0 (socket says 172.21.15.102:47170) 2024-03-30T13:57:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 -- 172.21.15.102:0/2708592172 learned_addr learned my addr 172.21.15.102:0/2708592172 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:34.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573e59c700 1 --2- 172.21.15.102:0/2708592172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5740101c90 0x7f5740198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 -- 172.21.15.102:0/2708592172 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740102650 msgr2=0x7f57401986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 --2- 172.21.15.102:0/2708592172 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740102650 0x7f57401986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 -- 172.21.15.102:0/2708592172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5740101c90 msgr2=0x7f5740198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 --2- 172.21.15.102:0/2708592172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5740101c90 0x7f5740198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 -- 172.21.15.102:0/2708592172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574019d180 con 0x7f5740106180 2024-03-30T13:57:34.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573dd9b700 1 --2- 172.21.15.102:0/2708592172 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5740102650 0x7f57401986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:34.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f573ed9d700 1 --2- 172.21.15.102:0/2708592172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106180 0x7f574019ca40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5738007d70 tx=0x7f573800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:34.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738016020 con 0x7f5740106180 2024-03-30T13:57:34.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574019d470 con 0x7f5740106180 2024-03-30T13:57:34.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57401a4cf0 con 0x7f5740106180 2024-03-30T13:57:34.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f573800ac90 con 0x7f5740106180 2024-03-30T13:57:34.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.595+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57380115e0 con 0x7f5740106180 2024-03-30T13:57:34.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.596+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5720000f80 con 0x7f5740106180 2024-03-30T13:57:34.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.597+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f5738019020 con 0x7f5740106180 2024-03-30T13:57:34.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.597+0000 7f572f7fe700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6800/622906077 conn(0x7f572806dc00 0x7f57280700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.597+0000 7f573e59c700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6800/622906077 conn(0x7f572806dc00 0x7f57280700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.597+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f573808dec0 con 0x7f5740106180 2024-03-30T13:57:34.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.598+0000 7f572f7fe700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6801/3100565996 conn(0x7f57280774a0 0x7f5728079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:34.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.598+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 --> v2:172.21.15.102:6801/3100565996 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f573801ce50 con 0x7f57280774a0 2024-03-30T13:57:34.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.598+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f573808e350 con 0x7f5740106180 2024-03-30T13:57:34.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.598+0000 7f573dd9b700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6801/3100565996 conn(0x7f57280774a0 0x7f5728079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:34.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.598+0000 7f573e59c700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6800/622906077 conn(0x7f572806dc00 0x7f57280700c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5730009680 tx=0x7f5730009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:34.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.598+0000 7f573dd9b700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6801/3100565996 conn(0x7f57280774a0 0x7f5728079940 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:34.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.599+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== osd.0 v2:172.21.15.102:6801/3100565996 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f573801ce50 con 0x7f57280774a0 2024-03-30T13:57:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.614+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 --> v2:172.21.15.102:6801/3100565996 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5720002d00 con 0x7f57280774a0 2024-03-30T13:57:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f572f7fe700 1 -- 172.21.15.102:0/2708592172 <== osd.0 v2:172.21.15.102:6801/3100565996 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5720002d00 con 0x7f57280774a0 2024-03-30T13:57:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6801/3100565996 conn(0x7f57280774a0 msgr2=0x7f5728079940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6801/3100565996 conn(0x7f57280774a0 0x7f5728079940 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6800/622906077 conn(0x7f572806dc00 msgr2=0x7f57280700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 --2- 172.21.15.102:0/2708592172 >> v2:172.21.15.102:6800/622906077 conn(0x7f572806dc00 0x7f57280700c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5730009680 tx=0x7f5730009260 comp rx=0 tx=0).stop 2024-03-30T13:57:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106180 msgr2=0x7f574019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:34.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 --2- 172.21.15.102:0/2708592172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106180 0x7f574019ca40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5738007d70 tx=0x7f573800a340 comp rx=0 tx=0).stop 2024-03-30T13:57:34.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f573ed9d700 1 -- 172.21.15.102:0/2708592172 reap_dead start 2024-03-30T13:57:34.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 shutdown_connections 2024-03-30T13:57:34.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 >> 172.21.15.102:0/2708592172 conn(0x7f5740076480 msgr2=0x7f5740100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:34.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 shutdown_connections 2024-03-30T13:57:34.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:34.615+0000 7f5744d67700 1 -- 172.21.15.102:0/2708592172 wait complete. 2024-03-30T13:57:35.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:57:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:57:35.698 INFO:teuthology.orchestra.run.smithi102.stdout:34359738418 2024-03-30T13:57:35.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T13:57:36.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:35 smithi138 ceph-mon[33397]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-30T13:57:36.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:35 smithi102 ceph-mon[29392]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-30T13:57:36.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:35 smithi102 ceph-mon[37717]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-30T13:57:36.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:57:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:37.266 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:37.267 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:37.292 INFO:teuthology.orchestra.run.smithi102.stdout:55834574893 2024-03-30T13:57:37.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T13:57:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 -- 172.21.15.102:0/2169813774 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 msgr2=0x7f3a4c102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 --2- 172.21.15.102:0/2169813774 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 0x7f3a4c102710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3a380098f0 tx=0x7f3a38009c00 comp rx=0 tx=0).stop 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 -- 172.21.15.102:0/2169813774 shutdown_connections 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 --2- 172.21.15.102:0/2169813774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a4c108270 0x7f3a4c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 --2- 172.21.15.102:0/2169813774 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a4c102c50 0x7f3a4c10f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 --2- 172.21.15.102:0/2169813774 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 0x7f3a4c102710 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 -- 172.21.15.102:0/2169813774 >> 172.21.15.102:0/2169813774 conn(0x7f3a4c079370 msgr2=0x7f3a4c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 -- 172.21.15.102:0/2169813774 shutdown_connections 2024-03-30T13:57:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.509+0000 7f3a52e6a700 1 -- 172.21.15.102:0/2169813774 wait complete. 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 Processor -- start 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 -- start start 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 0x7f3a4c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a4c102c50 0x7f3a4c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a4c108270 0x7f3a4c19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c111bf0 con 0x7f3a4c102c50 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c111a70 con 0x7f3a4c108270 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a52e6a700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4c111d70 con 0x7f3a4c102290 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 0x7f3a4c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 0x7f3a4c1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:47206/0 (socket says 172.21.15.102:47206) 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 -- 172.21.15.102:0/1158943339 learned_addr learned my addr 172.21.15.102:0/1158943339 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a4bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a4c102c50 0x7f3a4c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a51407700 1 --2- 172.21.15.102:0/1158943339 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a4c108270 0x7f3a4c19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 -- 172.21.15.102:0/1158943339 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a4c108270 msgr2=0x7f3a4c19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 --2- 172.21.15.102:0/1158943339 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a4c108270 0x7f3a4c19c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 -- 172.21.15.102:0/1158943339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a4c102c50 msgr2=0x7f3a4c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 --2- 172.21.15.102:0/1158943339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a4c102c50 0x7f3a4c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.510+0000 7f3a50c06700 1 -- 172.21.15.102:0/1158943339 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c19d0b0 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.511+0000 7f3a50c06700 1 --2- 172.21.15.102:0/1158943339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 0x7f3a4c1980f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000ec10 tx=0x7f3a4000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.511+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4000cd40 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.511+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f3a400044d0 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.511+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c19d3a0 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.511+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40010690 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.511+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c1a4c20 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.512+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3a4c107790 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.512+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f3a40004020 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a49ffb700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6800/622906077 conn(0x7f3a3c06dc00 0x7f3a3c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f3a4008de30 con 0x7f3a4c102290 2024-03-30T13:57:37.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a4bfff700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6800/622906077 conn(0x7f3a3c06dc00 0x7f3a3c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a49ffb700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6813/1080330048 conn(0x7f3a3c0774a0 0x7f3a3c079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 --> v2:172.21.15.102:6813/1080330048 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a3c079fd0 con 0x7f3a3c0774a0 2024-03-30T13:57:37.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f3a40090530 con 0x7f3a4c102290 2024-03-30T13:57:37.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a51407700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6813/1080330048 conn(0x7f3a3c0774a0 0x7f3a3c079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a4bfff700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6800/622906077 conn(0x7f3a3c06dc00 0x7f3a3c0700c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009fd0 tx=0x7f3a38019040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:37.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.513+0000 7f3a51407700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6813/1080330048 conn(0x7f3a3c0774a0 0x7f3a3c079940 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:37.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.514+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== osd.3 v2:172.21.15.102:6813/1080330048 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3a3c079fd0 con 0x7f3a3c0774a0 2024-03-30T13:57:37.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.530+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 --> v2:172.21.15.102:6813/1080330048 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3a4c103920 con 0x7f3a3c0774a0 2024-03-30T13:57:37.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.530+0000 7f3a49ffb700 1 -- 172.21.15.102:0/1158943339 <== osd.3 v2:172.21.15.102:6813/1080330048 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3a4c103920 con 0x7f3a3c0774a0 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6813/1080330048 conn(0x7f3a3c0774a0 msgr2=0x7f3a3c079940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6813/1080330048 conn(0x7f3a3c0774a0 0x7f3a3c079940 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6800/622906077 conn(0x7f3a3c06dc00 msgr2=0x7f3a3c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 --2- 172.21.15.102:0/1158943339 >> v2:172.21.15.102:6800/622906077 conn(0x7f3a3c06dc00 0x7f3a3c0700c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009fd0 tx=0x7f3a38019040 comp rx=0 tx=0).stop 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 msgr2=0x7f3a4c1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 --2- 172.21.15.102:0/1158943339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3a4c102290 0x7f3a4c1980f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000ec10 tx=0x7f3a4000ef20 comp rx=0 tx=0).stop 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a51407700 1 -- 172.21.15.102:0/1158943339 reap_dead start 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 shutdown_connections 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 >> 172.21.15.102:0/1158943339 conn(0x7f3a4c079370 msgr2=0x7f3a4c105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:37.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 shutdown_connections 2024-03-30T13:57:37.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:37.531+0000 7f3a52e6a700 1 -- 172.21.15.102:0/1158943339 wait complete. 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 -- 172.21.15.102:0/2100727534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 msgr2=0x7f6b4810f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 --2- 172.21.15.102:0/2100727534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 0x7f6b4810f150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009fc0 tx=0x7f6b4000b330 comp rx=0 tx=0).stop 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 -- 172.21.15.102:0/2100727534 shutdown_connections 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 --2- 172.21.15.102:0/2100727534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 0x7f6b4810f150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 --2- 172.21.15.102:0/2100727534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b481024a0 0x7f6b48102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 --2- 172.21.15.102:0/2100727534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b48102e50 0x7f6b48106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 -- 172.21.15.102:0/2100727534 >> 172.21.15.102:0/2100727534 conn(0x7f6b480fdd10 msgr2=0x7f6b48100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 -- 172.21.15.102:0/2100727534 shutdown_connections 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.890+0000 7f6b4e748700 1 -- 172.21.15.102:0/2100727534 wait complete. 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 Processor -- start 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 -- start start 2024-03-30T13:57:37.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b481024a0 0x7f6b48198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b48102e50 0x7f6b48198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 0x7f6b4819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48111c70 con 0x7f6b48107580 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48111af0 con 0x7f6b48102e50 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4e748700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6b48111df0 con 0x7f6b481024a0 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b47fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b481024a0 0x7f6b48198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4cce5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 0x7f6b4819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b47fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b481024a0 0x7f6b48198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:47192/0 (socket says 172.21.15.102:47192) 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b47fff700 1 -- 172.21.15.102:0/1274688545 learned_addr learned my addr 172.21.15.102:0/1274688545 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b477fe700 1 --2- 172.21.15.102:0/1274688545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b48102e50 0x7f6b48198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4cce5700 1 -- 172.21.15.102:0/1274688545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b481024a0 msgr2=0x7f6b48198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4cce5700 1 --2- 172.21.15.102:0/1274688545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b481024a0 0x7f6b48198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4cce5700 1 -- 172.21.15.102:0/1274688545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b48102e50 msgr2=0x7f6b48198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4cce5700 1 --2- 172.21.15.102:0/1274688545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b48102e50 0x7f6b48198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b4cce5700 1 -- 172.21.15.102:0/1274688545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4819d0f0 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.891+0000 7f6b47fff700 1 --2- 172.21.15.102:0/1274688545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b481024a0 0x7f6b48198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.892+0000 7f6b4cce5700 1 --2- 172.21.15.102:0/1274688545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 0x7f6b4819c9b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6b40015000 tx=0x7f6b4000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.892+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40022070 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.892+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f6b4001c6c0 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.892+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4819d380 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.892+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40005380 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.892+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b48074470 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.892+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6b480fead0 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.893+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f6b4002d020 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b457fa700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.102:6800/622906077 conn(0x7f6b3006dc00 0x7f6b300700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b47fff700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.102:6800/622906077 conn(0x7f6b3006dc00 0x7f6b300700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f6b4000f020 con 0x7f6b48107580 2024-03-30T13:57:37.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b457fa700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.138:6800/1718735591 conn(0x7f6b300774a0 0x7f6b30079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 --> v2:172.21.15.138:6800/1718735591 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b40025e70 con 0x7f6b300774a0 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f6b400930d0 con 0x7f6b48107580 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b47fff700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.102:6800/622906077 conn(0x7f6b3006dc00 0x7f6b300700c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b38005950 tx=0x7f6b3800b500 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b477fe700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.138:6800/1718735591 conn(0x7f6b300774a0 0x7f6b30079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.894+0000 7f6b477fe700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.138:6800/1718735591 conn(0x7f6b300774a0 0x7f6b30079940 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.895+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== osd.4 v2:172.21.15.138:6800/1718735591 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b40025e70 con 0x7f6b300774a0 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.911+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 --> v2:172.21.15.138:6800/1718735591 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6b48103980 con 0x7f6b300774a0 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b457fa700 1 -- 172.21.15.102:0/1274688545 <== osd.4 v2:172.21.15.138:6800/1718735591 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6b48103980 con 0x7f6b300774a0 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 >> v2:172.21.15.138:6800/1718735591 conn(0x7f6b300774a0 msgr2=0x7f6b30079940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.138:6800/1718735591 conn(0x7f6b300774a0 0x7f6b30079940 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 >> v2:172.21.15.102:6800/622906077 conn(0x7f6b3006dc00 msgr2=0x7f6b300700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 --2- 172.21.15.102:0/1274688545 >> v2:172.21.15.102:6800/622906077 conn(0x7f6b3006dc00 0x7f6b300700c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b38005950 tx=0x7f6b3800b500 comp rx=0 tx=0).stop 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 msgr2=0x7f6b4819c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 --2- 172.21.15.102:0/1274688545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b48107580 0x7f6b4819c9b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6b40015000 tx=0x7f6b4000ccb0 comp rx=0 tx=0).stop 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4cce5700 1 -- 172.21.15.102:0/1274688545 reap_dead start 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 shutdown_connections 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 >> 172.21.15.102:0/1274688545 conn(0x7f6b480fdd10 msgr2=0x7f6b480fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 shutdown_connections 2024-03-30T13:57:37.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:35.912+0000 7f6b4e748700 1 -- 172.21.15.102:0/1274688545 wait complete. 2024-03-30T13:57:38.181 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:37 smithi102 ceph-mon[29392]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-30T13:57:38.181 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:37 smithi102 ceph-mon[37717]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-30T13:57:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:37 smithi138 ceph-mon[33397]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-30T13:57:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 -- 172.21.15.102:0/423147943 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 msgr2=0x7f6e04107560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 --2- 172.21.15.102:0/423147943 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 0x7f6e04107560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009990 tx=0x7f6e00009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 -- 172.21.15.102:0/423147943 shutdown_connections 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 --2- 172.21.15.102:0/423147943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e04101ae0 0x7f6e0410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 --2- 172.21.15.102:0/423147943 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 0x7f6e04107560 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 --2- 172.21.15.102:0/423147943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6e04101140 0x7f6e041015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 -- 172.21.15.102:0/423147943 >> 172.21.15.102:0/423147943 conn(0x7f6e04078070 msgr2=0x7f6e04078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 -- 172.21.15.102:0/423147943 shutdown_connections 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.561+0000 7f6e0c535700 1 -- 172.21.15.102:0/423147943 wait complete. 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 Processor -- start 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 -- start start 2024-03-30T13:57:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6e04101140 0x7f6e041981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e04101ae0 0x7f6e041986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 0x7f6e0419ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6e04110b40 con 0x7f6e04101ae0 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6e041109c0 con 0x7f6e04101140 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0c535700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6e04110cc0 con 0x7f6e04107180 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0aad2700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 0x7f6e0419ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e09ad0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e04101ae0 0x7f6e041986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0aad2700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 0x7f6e0419ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:47234/0 (socket says 172.21.15.102:47234) 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0aad2700 1 -- 172.21.15.102:0/2733851108 learned_addr learned my addr 172.21.15.102:0/2733851108 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.562+0000 7f6e0a2d1700 1 --2- 172.21.15.102:0/2733851108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6e04101140 0x7f6e041981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e09ad0700 1 -- 172.21.15.102:0/2733851108 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 msgr2=0x7f6e0419ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e09ad0700 1 --2- 172.21.15.102:0/2733851108 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6e04107180 0x7f6e0419ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e09ad0700 1 -- 172.21.15.102:0/2733851108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6e04101140 msgr2=0x7f6e041981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e09ad0700 1 --2- 172.21.15.102:0/2733851108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6e04101140 0x7f6e041981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e09ad0700 1 -- 172.21.15.102:0/2733851108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0419d1b0 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e09ad0700 1 --2- 172.21.15.102:0/2733851108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e04101ae0 0x7f6e041986e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6df400b450 tx=0x7f6df400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4015020 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0419d4a0 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f6df4011ce0 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4010c00 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.563+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e041a4d20 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.564+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e0406a8d0 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.564+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f6df4018020 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6dfb7fe700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6800/622906077 conn(0x7f6df006dc00 0x7f6df00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f6df408d6d0 con 0x7f6e04101ae0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6e0a2d1700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6800/622906077 conn(0x7f6df006dc00 0x7f6df00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6dfb7fe700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6809/203063794 conn(0x7f6df00774a0 0x7f6df0079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 --> v2:172.21.15.102:6809/203063794 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6df4018ec0 con 0x7f6df00774a0 2024-03-30T13:57:38.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f6df408db60 con 0x7f6e04101ae0 2024-03-30T13:57:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6e0aad2700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6809/203063794 conn(0x7f6df00774a0 0x7f6df0079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6e0a2d1700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6800/622906077 conn(0x7f6df006dc00 0x7f6df00700c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b2e0 tx=0x7f6e00005fd0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.565+0000 7f6e0aad2700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6809/203063794 conn(0x7f6df00774a0 0x7f6df0079940 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.566+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== osd.2 v2:172.21.15.102:6809/203063794 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6df4018ec0 con 0x7f6df00774a0 2024-03-30T13:57:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.582+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 --> v2:172.21.15.102:6809/203063794 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6e04102830 con 0x7f6df00774a0 2024-03-30T13:57:38.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6dfb7fe700 1 -- 172.21.15.102:0/2733851108 <== osd.2 v2:172.21.15.102:6809/203063794 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6e04102830 con 0x7f6df00774a0 2024-03-30T13:57:38.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6809/203063794 conn(0x7f6df00774a0 msgr2=0x7f6df0079940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:38.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6809/203063794 conn(0x7f6df00774a0 0x7f6df0079940 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6800/622906077 conn(0x7f6df006dc00 msgr2=0x7f6df00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:38.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 --2- 172.21.15.102:0/2733851108 >> v2:172.21.15.102:6800/622906077 conn(0x7f6df006dc00 0x7f6df00700c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b2e0 tx=0x7f6e00005fd0 comp rx=0 tx=0).stop 2024-03-30T13:57:38.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e04101ae0 msgr2=0x7f6e041986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:38.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 --2- 172.21.15.102:0/2733851108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e04101ae0 0x7f6e041986e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6df400b450 tx=0x7f6df400b760 comp rx=0 tx=0).stop 2024-03-30T13:57:38.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0aad2700 1 -- 172.21.15.102:0/2733851108 reap_dead start 2024-03-30T13:57:38.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 shutdown_connections 2024-03-30T13:57:38.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 >> 172.21.15.102:0/2733851108 conn(0x7f6e04078070 msgr2=0x7f6e04069ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:38.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 shutdown_connections 2024-03-30T13:57:38.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:38.583+0000 7f6e0c535700 1 -- 172.21.15.102:0/2733851108 wait complete. 2024-03-30T13:57:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:163208757271 2024-03-30T13:57:39.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T13:57:39.828 INFO:teuthology.orchestra.run.smithi102.stdout:107374182436 2024-03-30T13:57:39.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T13:57:40.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:39 smithi102 ceph-mon[29392]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-30T13:57:40.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:39 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:39 smithi102 ceph-mon[37717]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-30T13:57:40.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:39 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.435+0000 7fd26f35e700 1 -- 172.21.15.102:0/2407299928 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 msgr2=0x7fd268106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.435+0000 7fd26f35e700 1 --2- 172.21.15.102:0/2407299928 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 0x7fd268106e90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2580099c0 tx=0x7fd258009cd0 comp rx=0 tx=0).stop 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.435+0000 7fd26f35e700 1 -- 172.21.15.102:0/2407299928 shutdown_connections 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.435+0000 7fd26f35e700 1 --2- 172.21.15.102:0/2407299928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd26806ac00 0x7fd26806afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.435+0000 7fd26f35e700 1 --2- 172.21.15.102:0/2407299928 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2681075c0 0x7fd26810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.435+0000 7fd26f35e700 1 --2- 172.21.15.102:0/2407299928 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 0x7fd268106e90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.435+0000 7fd26f35e700 1 -- 172.21.15.102:0/2407299928 >> 172.21.15.102:0/2407299928 conn(0x7fd268077150 msgr2=0x7fd268077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 -- 172.21.15.102:0/2407299928 shutdown_connections 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 -- 172.21.15.102:0/2407299928 wait complete. 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 Processor -- start 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 -- start start 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd26806ac00 0x7fd268101240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 0x7fd268101780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2681075c0 0x7fd268101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd268111c60 con 0x7fd26806ac00 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd268111ae0 con 0x7fd2681075c0 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.436+0000 7fd26f35e700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd268111de0 con 0x7fd26806b5b0 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 0x7fd268101780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26d0fa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd26806ac00 0x7fd268101240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 0x7fd268101780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:47256/0 (socket says 172.21.15.102:47256) 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 -- 172.21.15.102:0/344600406 learned_addr learned my addr 172.21.15.102:0/344600406 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:40.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26d8fb700 1 --2- 172.21.15.102:0/344600406 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2681075c0 0x7fd268101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 -- 172.21.15.102:0/344600406 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2681075c0 msgr2=0x7fd268101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 --2- 172.21.15.102:0/344600406 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2681075c0 0x7fd268101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 -- 172.21.15.102:0/344600406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd26806ac00 msgr2=0x7fd268101240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 --2- 172.21.15.102:0/344600406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd26806ac00 0x7fd268101240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 -- 172.21.15.102:0/344600406 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2681a6630 con 0x7fd26806b5b0 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26d0fa700 1 --2- 172.21.15.102:0/344600406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd26806ac00 0x7fd268101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26d8fb700 1 --2- 172.21.15.102:0/344600406 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2681075c0 0x7fd268101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26c8f9700 1 --2- 172.21.15.102:0/344600406 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 0x7fd268101780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2580098e0 tx=0x7fd25800f670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd258027070 con 0x7fd26806b5b0 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2681a68c0 con 0x7fd26806b5b0 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fd25800fb90 con 0x7fd26806b5b0 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd258022e50 con 0x7fd26806b5b0 2024-03-30T13:57:40.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.437+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2681a6d20 con 0x7fd26806b5b0 2024-03-30T13:57:40.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:39 smithi138 ceph-mon[33397]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-30T13:57:40.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:39 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.438+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2680ff9c0 con 0x7fd26806b5b0 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.439+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fd25802a070 con 0x7fd26806b5b0 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.439+0000 7fd25e7fc700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.102:6800/622906077 conn(0x7fd25406dda0 0x7fd254070260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.439+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7fd25808eee0 con 0x7fd26806b5b0 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.439+0000 7fd26d0fa700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.102:6800/622906077 conn(0x7fd25406dda0 0x7fd254070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.439+0000 7fd25e7fc700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.138:6812/3765635388 conn(0x7fd254077640 0x7fd254079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.439+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 --> v2:172.21.15.138:6812/3765635388 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd258022e50 con 0x7fd254077640 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.439+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fd25808f370 con 0x7fd26806b5b0 2024-03-30T13:57:40.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.440+0000 7fd26d8fb700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.138:6812/3765635388 conn(0x7fd254077640 0x7fd254079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:40.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.440+0000 7fd26d0fa700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.102:6800/622906077 conn(0x7fd25406dda0 0x7fd254070260 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd264009680 tx=0x7fd264009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:40.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.440+0000 7fd26d8fb700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.138:6812/3765635388 conn(0x7fd254077640 0x7fd254079ae0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:40.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.441+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== osd.7 v2:172.21.15.138:6812/3765635388 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd258022e50 con 0x7fd254077640 2024-03-30T13:57:40.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.470+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 --> v2:172.21.15.138:6812/3765635388 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd268103970 con 0x7fd254077640 2024-03-30T13:57:40.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.470+0000 7fd25e7fc700 1 -- 172.21.15.102:0/344600406 <== osd.7 v2:172.21.15.138:6812/3765635388 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd268103970 con 0x7fd254077640 2024-03-30T13:57:40.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 >> v2:172.21.15.138:6812/3765635388 conn(0x7fd254077640 msgr2=0x7fd254079ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:40.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.138:6812/3765635388 conn(0x7fd254077640 0x7fd254079ae0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:40.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 >> v2:172.21.15.102:6800/622906077 conn(0x7fd25406dda0 msgr2=0x7fd254070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:40.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 --2- 172.21.15.102:0/344600406 >> v2:172.21.15.102:6800/622906077 conn(0x7fd25406dda0 0x7fd254070260 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd264009680 tx=0x7fd264009260 comp rx=0 tx=0).stop 2024-03-30T13:57:40.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 msgr2=0x7fd268101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:40.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 --2- 172.21.15.102:0/344600406 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd26806b5b0 0x7fd268101780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2580098e0 tx=0x7fd25800f670 comp rx=0 tx=0).stop 2024-03-30T13:57:40.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26d8fb700 1 -- 172.21.15.102:0/344600406 reap_dead start 2024-03-30T13:57:40.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 shutdown_connections 2024-03-30T13:57:40.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 >> 172.21.15.102:0/344600406 conn(0x7fd268077150 msgr2=0x7fd2680fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:40.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 shutdown_connections 2024-03-30T13:57:40.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:40.471+0000 7fd26f35e700 1 -- 172.21.15.102:0/344600406 wait complete. 2024-03-30T13:57:40.918 INFO:teuthology.orchestra.run.smithi102.stdout:128849018910 2024-03-30T13:57:40.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T13:57:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 -- 172.21.15.102:0/2020244006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102ed0 msgr2=0x7f5428106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 --2- 172.21.15.102:0/2020244006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102ed0 0x7f5428106ea0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f54180099c0 tx=0x7f5418009cd0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 -- 172.21.15.102:0/2020244006 shutdown_connections 2024-03-30T13:57:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 --2- 172.21.15.102:0/2020244006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54281075d0 0x7f542810f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 --2- 172.21.15.102:0/2020244006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5428102520 0x7f5428102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 --2- 172.21.15.102:0/2020244006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102ed0 0x7f5428106ea0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 -- 172.21.15.102:0/2020244006 >> 172.21.15.102:0/2020244006 conn(0x7f5428079360 msgr2=0x7f5428079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 -- 172.21.15.102:0/2020244006 shutdown_connections 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.948+0000 7f5430474700 1 -- 172.21.15.102:0/2020244006 wait complete. 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 Processor -- start 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 -- start start 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102520 0x7f54281981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5428102ed0 0x7f5428198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54281075d0 0x7f542819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5428111c40 con 0x7f54281075d0 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5428111ac0 con 0x7f5428102ed0 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f5430474700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5428111dc0 con 0x7f5428102520 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542e210700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102520 0x7f54281981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542ea11700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54281075d0 0x7f542819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542e210700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102520 0x7f54281981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:37208/0 (socket says 172.21.15.102:37208) 2024-03-30T13:57:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542ea11700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54281075d0 0x7f542819c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59308/0 (socket says 172.21.15.102:59308) 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542e210700 1 -- 172.21.15.102:0/3719731176 learned_addr learned my addr 172.21.15.102:0/3719731176 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542da0f700 1 --2- 172.21.15.102:0/3719731176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5428102ed0 0x7f5428198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542ea11700 1 -- 172.21.15.102:0/3719731176 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102520 msgr2=0x7f54281981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542ea11700 1 --2- 172.21.15.102:0/3719731176 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102520 0x7f54281981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542ea11700 1 -- 172.21.15.102:0/3719731176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5428102ed0 msgr2=0x7f5428198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542ea11700 1 --2- 172.21.15.102:0/3719731176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5428102ed0 0x7f5428198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542ea11700 1 -- 172.21.15.102:0/3719731176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542819d130 con 0x7f54281075d0 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542da0f700 1 --2- 172.21.15.102:0/3719731176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5428102ed0 0x7f5428198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.949+0000 7f542e210700 1 --2- 172.21.15.102:0/3719731176 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5428102520 0x7f54281981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.950+0000 7f542ea11700 1 --2- 172.21.15.102:0/3719731176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54281075d0 0x7f542819c9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f542000efb0 tx=0x7f542000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.950+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542000ca50 con 0x7f54281075d0 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.950+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f542000cbf0 con 0x7f54281075d0 2024-03-30T13:57:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.950+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542819d420 con 0x7f54281075d0 2024-03-30T13:57:41.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.950+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420004850 con 0x7f54281075d0 2024-03-30T13:57:41.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.950+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54281a4ca0 con 0x7f54281075d0 2024-03-30T13:57:41.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.951+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5428100ad0 con 0x7f54281075d0 2024-03-30T13:57:41.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.951+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f5420012050 con 0x7f54281075d0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f541f7fe700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.102:6800/622906077 conn(0x7f541406dc00 0x7f54140700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f542e210700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.102:6800/622906077 conn(0x7f541406dc00 0x7f54140700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f542008d4c0 con 0x7f54281075d0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f541f7fe700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.138:6808/3214961690 conn(0x7f54140774a0 0x7f5414079940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 --> v2:172.21.15.138:6808/3214961690 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f542000ae50 con 0x7f54140774a0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f5420094050 con 0x7f54281075d0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f542e210700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.102:6800/622906077 conn(0x7f541406dc00 0x7f54140700c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f54240105c0 tx=0x7f542400b480 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.952+0000 7f542da0f700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.138:6808/3214961690 conn(0x7f54140774a0 0x7f5414079940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.953+0000 7f542da0f700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.138:6808/3214961690 conn(0x7f54140774a0 0x7f5414079940 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:41.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.954+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== osd.6 v2:172.21.15.138:6808/3214961690 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f542000ae50 con 0x7f54140774a0 2024-03-30T13:57:41.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.973+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 --> v2:172.21.15.138:6808/3214961690 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5428103980 con 0x7f54140774a0 2024-03-30T13:57:41.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.973+0000 7f541f7fe700 1 -- 172.21.15.102:0/3719731176 <== osd.6 v2:172.21.15.138:6808/3214961690 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5428103980 con 0x7f54140774a0 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 >> v2:172.21.15.138:6808/3214961690 conn(0x7f54140774a0 msgr2=0x7f5414079940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.138:6808/3214961690 conn(0x7f54140774a0 0x7f5414079940 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 >> v2:172.21.15.102:6800/622906077 conn(0x7f541406dc00 msgr2=0x7f54140700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 --2- 172.21.15.102:0/3719731176 >> v2:172.21.15.102:6800/622906077 conn(0x7f541406dc00 0x7f54140700c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f54240105c0 tx=0x7f542400b480 comp rx=0 tx=0).stop 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54281075d0 msgr2=0x7f542819c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 --2- 172.21.15.102:0/3719731176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54281075d0 0x7f542819c9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f542000efb0 tx=0x7f542000c3a0 comp rx=0 tx=0).stop 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f542ea11700 1 -- 172.21.15.102:0/3719731176 reap_dead start 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 shutdown_connections 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 >> 172.21.15.102:0/3719731176 conn(0x7f5428079360 msgr2=0x7f5428100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:41.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 shutdown_connections 2024-03-30T13:57:41.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:41.974+0000 7f5430474700 1 -- 172.21.15.102:0/3719731176 wait complete. 2024-03-30T13:57:42.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:41 smithi102 ceph-mon[29392]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-30T13:57:42.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:41 smithi102 ceph-mon[37717]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-30T13:57:42.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:41 smithi138 ceph-mon[33397]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-30T13:57:42.605 INFO:teuthology.orchestra.run.smithi102.stdout:197568495637 2024-03-30T13:57:42.606 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T13:57:43.218 INFO:teuthology.orchestra.run.smithi102.stdout:231928234000 2024-03-30T13:57:43.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T13:57:43.668 INFO:teuthology.orchestra.run.smithi102.stdout:77309411369 2024-03-30T13:57:43.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T13:57:43.729 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:43.730 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:43.781 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:44.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:43 smithi102 ceph-mon[29392]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:57:44.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:43 smithi102 ceph-mon[37717]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:57:44.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:43 smithi138 ceph-mon[33397]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:57:45.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:57:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:57:46.278 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:46.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:45 smithi138 ceph-mon[33397]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:46.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:45 smithi102 ceph-mon[37717]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:46.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:45 smithi102 ceph-mon[29392]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:46.688 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:57:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:47.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:47.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:47 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:47.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:47 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:47.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:47.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:47 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:47.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:47 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:47.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:47.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:47 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:47.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:47 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:48.232 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:48 smithi102 ceph-mon[37717]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:48.232 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:48 smithi102 ceph-mon[29392]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:48.336 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:48 smithi138 ceph-mon[33397]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:48.551 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 -- 172.21.15.102:0/3347022657 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 msgr2=0x7fd744107560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 --2- 172.21.15.102:0/3347022657 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 0x7fd744107560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd734009990 tx=0x7fd734009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 -- 172.21.15.102:0/3347022657 shutdown_connections 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 --2- 172.21.15.102:0/3347022657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd744101ae0 0x7fd74410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 --2- 172.21.15.102:0/3347022657 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 0x7fd744107560 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 --2- 172.21.15.102:0/3347022657 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101140 0x7fd7441015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 -- 172.21.15.102:0/3347022657 >> 172.21.15.102:0/3347022657 conn(0x7fd744078070 msgr2=0x7fd744078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 -- 172.21.15.102:0/3347022657 shutdown_connections 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.503+0000 7fd749a81700 1 -- 172.21.15.102:0/3347022657 wait complete. 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 Processor -- start 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 -- start start 2024-03-30T13:57:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd744101140 0x7fd744102df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101ae0 0x7fd744103330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 0x7fd744103870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd744110ae0 con 0x7fd744101140 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd744110960 con 0x7fd744101ae0 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd749a81700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd744110c60 con 0x7fd744107180 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd743fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 0x7fd744103870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd742ffd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101ae0 0x7fd744103330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd7437fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd744101140 0x7fd744102df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd743fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 0x7fd744103870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:37230/0 (socket says 172.21.15.102:37230) 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd742ffd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101ae0 0x7fd744103330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.102:34072/0 (socket says 172.21.15.102:34072) 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd743fff700 1 -- 172.21.15.102:0/1723315593 learned_addr learned my addr 172.21.15.102:0/1723315593 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.504+0000 7fd7437fe700 1 -- 172.21.15.102:0/1723315593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 msgr2=0x7fd744103870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd7437fe700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 0x7fd744103870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd7437fe700 1 -- 172.21.15.102:0/1723315593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101ae0 msgr2=0x7fd744103330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd7437fe700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101ae0 0x7fd744103330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd7437fe700 1 -- 172.21.15.102:0/1723315593 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7441a66e0 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd742ffd700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101ae0 0x7fd744103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd7437fe700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd744101140 0x7fd744102df0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd7340098e0 tx=0x7fd734009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd740ff9700 1 -- 172.21.15.102:0/1723315593 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734027070 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd740ff9700 1 -- 172.21.15.102:0/1723315593 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fd73400fca0 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7441a6970 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd740ff9700 1 -- 172.21.15.102:0/1723315593 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734022e00 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.505+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7441a6dc0 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.506+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd744068f00 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.506+0000 7fd740ff9700 1 -- 172.21.15.102:0/1723315593 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fd73402a070 con 0x7fd744101140 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.507+0000 7fd740ff9700 1 --2- 172.21.15.102:0/1723315593 >> v2:172.21.15.102:6800/622906077 conn(0x7fd72c06dc00 0x7fd72c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.507+0000 7fd740ff9700 1 -- 172.21.15.102:0/1723315593 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7fd73408e810 con 0x7fd744101140 2024-03-30T13:57:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.507+0000 7fd742ffd700 1 --2- 172.21.15.102:0/1723315593 >> v2:172.21.15.102:6800/622906077 conn(0x7fd72c06dc00 0x7fd72c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.507+0000 7fd742ffd700 1 --2- 172.21.15.102:0/1723315593 >> v2:172.21.15.102:6800/622906077 conn(0x7fd72c06dc00 0x7fd72c0700c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd744111530 tx=0x7fd738009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.512+0000 7fd740ff9700 1 -- 172.21.15.102:0/1723315593 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd734013070 con 0x7fd744101140 2024-03-30T13:57:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.723+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd7441a6f90 con 0x7fd744101140 2024-03-30T13:57:49.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.724+0000 7fd740ff9700 1 -- 172.21.15.102:0/1723315593 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd73405fd40 con 0x7fd744101140 2024-03-30T13:57:49.721 INFO:teuthology.orchestra.run.smithi102.stdout:231928234001 2024-03-30T13:57:49.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 >> v2:172.21.15.102:6800/622906077 conn(0x7fd72c06dc00 msgr2=0x7fd72c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 --2- 172.21.15.102:0/1723315593 >> v2:172.21.15.102:6800/622906077 conn(0x7fd72c06dc00 0x7fd72c0700c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd744111530 tx=0x7fd738009260 comp rx=0 tx=0).stop 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd744101140 msgr2=0x7fd744102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd744101140 0x7fd744102df0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd7340098e0 tx=0x7fd734009550 comp rx=0 tx=0).stop 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 shutdown_connections 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd744101140 0x7fd744102df0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 --2- 172.21.15.102:0/1723315593 >> v2:172.21.15.102:6800/622906077 conn(0x7fd72c06dc00 0x7fd72c0700c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd744107180 0x7fd744103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 --2- 172.21.15.102:0/1723315593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd744101ae0 0x7fd744103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.726+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 >> 172.21.15.102:0/1723315593 conn(0x7fd744078070 msgr2=0x7fd7440ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:49.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.727+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 shutdown_connections 2024-03-30T13:57:49.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:49.727+0000 7fd749a81700 1 -- 172.21.15.102:0/1723315593 wait complete. 2024-03-30T13:57:50.357 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:50 smithi102 ceph-mon[37717]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:50.357 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1723315593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T13:57:50.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:50 smithi102 ceph-mon[29392]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:50.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1723315593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T13:57:50.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:50 smithi138 ceph-mon[33397]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:50.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1723315593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T13:57:51.293 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:51.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1595142597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 msgr2=0x7f2f9c1110e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1595142597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 0x7f2f9c1110e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400b0a0 tx=0x7f2f9400b3b0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1595142597 shutdown_connections 2024-03-30T13:57:51.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1595142597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c069ef0 0x7f2f9c06a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1595142597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 0x7f2f9c1110e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1595142597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2f9c06a810 0x7f2f9c10e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1595142597 >> 172.21.15.102:0/1595142597 conn(0x7f2f9c078070 msgr2=0x7f2f9c078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:51.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1595142597 shutdown_connections 2024-03-30T13:57:51.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.313+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1595142597 wait complete. 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.314+0000 7f2fa2c6f700 1 Processor -- start 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.314+0000 7f2fa2c6f700 1 -- start start 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.314+0000 7f2fa2c6f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2f9c069ef0 0x7f2f9c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.314+0000 7f2fa2c6f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c06a810 0x7f2f9c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.314+0000 7f2fa2c6f700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 0x7f2f9c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa2c6f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c113db0 con 0x7f2f9c06a810 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa2c6f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c113c30 con 0x7f2f9c069ef0 2024-03-30T13:57:51.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa2c6f700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c113f30 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2f9bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c06a810 0x7f2f9c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2f9bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c06a810 0x7f2f9c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47056/0 (socket says 172.21.15.102:47056) 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2f9bfff700 1 -- 172.21.15.102:0/1377714919 learned_addr learned my addr 172.21.15.102:0/1377714919 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa120c700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 0x7f2f9c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa0a0b700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2f9c069ef0 0x7f2f9c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa120c700 1 -- 172.21.15.102:0/1377714919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2f9c069ef0 msgr2=0x7f2f9c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa120c700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2f9c069ef0 0x7f2f9c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa120c700 1 -- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c06a810 msgr2=0x7f2f9c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa120c700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c06a810 0x7f2f9c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.315+0000 7f2fa120c700 1 -- 172.21.15.102:0/1377714919 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c19f260 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.316+0000 7f2fa120c700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 0x7f2f9c19eb20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009bc0 tx=0x7f2f94004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.316+0000 7f2f99ffb700 1 -- 172.21.15.102:0/1377714919 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94021070 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.316+0000 7f2f99ffb700 1 -- 172.21.15.102:0/1377714919 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f2f94007b80 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.316+0000 7f2f99ffb700 1 -- 172.21.15.102:0/1377714919 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94005040 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.316+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9c19f4f0 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.316+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9c1a6dd0 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.317+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f9c068f00 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.318+0000 7f2f99ffb700 1 -- 172.21.15.102:0/1377714919 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f2f94024070 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.318+0000 7f2f99ffb700 1 --2- 172.21.15.102:0/1377714919 >> v2:172.21.15.102:6800/622906077 conn(0x7f2f8c06dc00 0x7f2f8c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.319+0000 7f2f99ffb700 1 -- 172.21.15.102:0/1377714919 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f2f9400f020 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.319+0000 7f2fa0a0b700 1 --2- 172.21.15.102:0/1377714919 >> v2:172.21.15.102:6800/622906077 conn(0x7f2f8c06dc00 0x7f2f8c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.319+0000 7f2fa0a0b700 1 --2- 172.21.15.102:0/1377714919 >> v2:172.21.15.102:6800/622906077 conn(0x7f2f8c06dc00 0x7f2f8c0700c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009680 tx=0x7f2f90009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:51.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.327+0000 7f2f99ffb700 1 -- 172.21.15.102:0/1377714919 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f9402b020 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.383 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.384 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:57:51.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:57:51.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.385 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.387 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:57:51.388 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:57:51.388 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.388 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.388 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.388 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.388 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T13:57:51.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:57:51.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:57:51.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:57:51.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.534+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2f9c1a6fa0 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.534+0000 7f2f99ffb700 1 -- 172.21.15.102:0/1377714919 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2f9405c9d0 con 0x7f2f9c10ecf0 2024-03-30T13:57:51.531 INFO:teuthology.orchestra.run.smithi102.stdout:107374182439 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 >> v2:172.21.15.102:6800/622906077 conn(0x7f2f8c06dc00 msgr2=0x7f2f8c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1377714919 >> v2:172.21.15.102:6800/622906077 conn(0x7f2f8c06dc00 0x7f2f8c0700c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009680 tx=0x7f2f90009260 comp rx=0 tx=0).stop 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 msgr2=0x7f2f9c19eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 0x7f2f9c19eb20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009bc0 tx=0x7f2f94004c80 comp rx=0 tx=0).stop 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 shutdown_connections 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1377714919 >> v2:172.21.15.102:6800/622906077 conn(0x7f2f8c06dc00 0x7f2f8c0700c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c10ecf0 0x7f2f9c19eb20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2f9c069ef0 0x7f2f9c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 --2- 172.21.15.102:0/1377714919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c06a810 0x7f2f9c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 >> 172.21.15.102:0/1377714919 conn(0x7f2f9c078070 msgr2=0x7f2f9c100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:51.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 shutdown_connections 2024-03-30T13:57:51.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.537+0000 7f2fa2c6f700 1 -- 172.21.15.102:0/1377714919 wait complete. 2024-03-30T13:57:51.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 -- 172.21.15.102:0/4049101088 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca201072b0 msgr2=0x7fca20107690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 --2- 172.21.15.102:0/4049101088 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca201072b0 0x7fca20107690 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0098f0 tx=0x7fca1c009c00 comp rx=0 tx=0).stop 2024-03-30T13:57:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 -- 172.21.15.102:0/4049101088 shutdown_connections 2024-03-30T13:57:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 --2- 172.21.15.102:0/4049101088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca201011f0 0x7fca2010f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 --2- 172.21.15.102:0/4049101088 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca201072b0 0x7fca20107690 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 --2- 172.21.15.102:0/4049101088 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca20100830 0x7fca20100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 -- 172.21.15.102:0/4049101088 >> 172.21.15.102:0/4049101088 conn(0x7fca20077740 msgr2=0x7fca200feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 -- 172.21.15.102:0/4049101088 shutdown_connections 2024-03-30T13:57:51.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.856+0000 7fca27e29700 1 -- 172.21.15.102:0/4049101088 wait complete. 2024-03-30T13:57:51.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.857+0000 7fca27e29700 1 Processor -- start 2024-03-30T13:57:51.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.857+0000 7fca27e29700 1 -- start start 2024-03-30T13:57:51.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.857+0000 7fca27e29700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca20100830 0x7fca201981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.857+0000 7fca27e29700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca201011f0 0x7fca20198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.857+0000 7fca27e29700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca201072b0 0x7fca2019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.857+0000 7fca263c6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca201072b0 0x7fca2019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.857+0000 7fca25bc5700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca20100830 0x7fca201981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca20100830 0x7fca201981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:35084/0 (socket says 172.21.15.102:35084) 2024-03-30T13:57:51.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca263c6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca201072b0 0x7fca2019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47102/0 (socket says 172.21.15.102:47102) 2024-03-30T13:57:51.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 -- 172.21.15.102:0/4003810947 learned_addr learned my addr 172.21.15.102:0/4003810947 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:51.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca253c4700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca201011f0 0x7fca20198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fca20111d30 con 0x7fca201072b0 2024-03-30T13:57:51.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fca20111bb0 con 0x7fca201011f0 2024-03-30T13:57:51.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fca20111eb0 con 0x7fca20100830 2024-03-30T13:57:51.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 -- 172.21.15.102:0/4003810947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca201011f0 msgr2=0x7fca20198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca201011f0 0x7fca20198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 -- 172.21.15.102:0/4003810947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca201072b0 msgr2=0x7fca2019caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:51.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca201072b0 0x7fca2019caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:51.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2019d1e0 con 0x7fca20100830 2024-03-30T13:57:51.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca25bc5700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca20100830 0x7fca201981d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0059d0 tx=0x7fca1c00fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:51.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca16ffd700 1 -- 172.21.15.102:0/4003810947 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c027070 con 0x7fca20100830 2024-03-30T13:57:51.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2019d470 con 0x7fca20100830 2024-03-30T13:57:51.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca201a4d50 con 0x7fca20100830 2024-03-30T13:57:51.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca16ffd700 1 -- 172.21.15.102:0/4003810947 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fca1c005570 con 0x7fca20100830 2024-03-30T13:57:51.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.858+0000 7fca16ffd700 1 -- 172.21.15.102:0/4003810947 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c004930 con 0x7fca20100830 2024-03-30T13:57:51.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.859+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca04005150 con 0x7fca20100830 2024-03-30T13:57:51.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.860+0000 7fca16ffd700 1 -- 172.21.15.102:0/4003810947 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fca1c02a070 con 0x7fca20100830 2024-03-30T13:57:51.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.861+0000 7fca16ffd700 1 --2- 172.21.15.102:0/4003810947 >> v2:172.21.15.102:6800/622906077 conn(0x7fca0c06dc00 0x7fca0c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:51.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.861+0000 7fca16ffd700 1 -- 172.21.15.102:0/4003810947 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7fca1c08e450 con 0x7fca20100830 2024-03-30T13:57:51.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.861+0000 7fca253c4700 1 --2- 172.21.15.102:0/4003810947 >> v2:172.21.15.102:6800/622906077 conn(0x7fca0c06dc00 0x7fca0c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:51.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.861+0000 7fca253c4700 1 --2- 172.21.15.102:0/4003810947 >> v2:172.21.15.102:6800/622906077 conn(0x7fca0c06dc00 0x7fca0c0700c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fca10009680 tx=0x7fca10009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:51.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.866+0000 7fca16ffd700 1 -- 172.21.15.102:0/4003810947 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fca1c013030 con 0x7fca20100830 2024-03-30T13:57:52.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.069+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fca04005710 con 0x7fca20100830 2024-03-30T13:57:52.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.070+0000 7fca16ffd700 1 -- 172.21.15.102:0/4003810947 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fca1c05bfb0 con 0x7fca20100830 2024-03-30T13:57:52.066 INFO:teuthology.orchestra.run.smithi102.stdout:197568495639 2024-03-30T13:57:52.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 >> v2:172.21.15.102:6800/622906077 conn(0x7fca0c06dc00 msgr2=0x7fca0c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:52.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 --2- 172.21.15.102:0/4003810947 >> v2:172.21.15.102:6800/622906077 conn(0x7fca0c06dc00 0x7fca0c0700c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fca10009680 tx=0x7fca10009260 comp rx=0 tx=0).stop 2024-03-30T13:57:52.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca20100830 msgr2=0x7fca201981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:52.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca20100830 0x7fca201981d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0059d0 tx=0x7fca1c00fa70 comp rx=0 tx=0).stop 2024-03-30T13:57:52.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 shutdown_connections 2024-03-30T13:57:52.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca201072b0 0x7fca2019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 --2- 172.21.15.102:0/4003810947 >> v2:172.21.15.102:6800/622906077 conn(0x7fca0c06dc00 0x7fca0c0700c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca20100830 0x7fca201981d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 --2- 172.21.15.102:0/4003810947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca201011f0 0x7fca20198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 >> 172.21.15.102:0/4003810947 conn(0x7fca20077740 msgr2=0x7fca20104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:52.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 shutdown_connections 2024-03-30T13:57:52.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:52.072+0000 7fca27e29700 1 -- 172.21.15.102:0/4003810947 wait complete. 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi102 to 3760M 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:57:52.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: Deploying daemon alertmanager.a on smithi102 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1377714919' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/270948129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4003810947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi102 to 3760M 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:57:52.396 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: Deploying daemon alertmanager.a on smithi102 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1377714919' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/270948129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T13:57:52.397 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4003810947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T13:57:52.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi102 to 3760M 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi138:/etc/ceph/ceph.conf 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Updating smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: Deploying daemon alertmanager.a on smithi102 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1377714919' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/270948129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T13:57:52.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4003810947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/564036339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 msgr2=0x7f3cf4102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/564036339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf4102770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009990 tx=0x7f3ce4009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/564036339 shutdown_connections 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/564036339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3cf4102cb0 0x7f3cf410f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/564036339 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cf4108360 0x7f3cf4108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/564036339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf4102770 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/564036339 >> 172.21.15.102:0/564036339 conn(0x7f3cf4079310 msgr2=0x7f3cf4079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/564036339 shutdown_connections 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/564036339 wait complete. 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 Processor -- start 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.753+0000 7f3cfa7f3700 1 -- start start 2024-03-30T13:57:52.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cfa7f3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cfa7f3700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3cf4102cb0 0x7f3cf419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cfa7f3700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cf4108360 0x7f3cf419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cfa7f3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4111cb0 con 0x7f3cf41022f0 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cfa7f3700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4111b30 con 0x7f3cf4108360 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cfa7f3700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf4111e30 con 0x7f3cf4102cb0 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf37fe700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3cf4102cb0 0x7f3cf419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47086/0 (socket says 172.21.15.102:47086) 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 -- 172.21.15.102:0/270948129 learned_addr learned my addr 172.21.15.102:0/270948129 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf8d90700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cf4108360 0x7f3cf419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 -- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3cf4102cb0 msgr2=0x7f3cf419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3cf4102cb0 0x7f3cf419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 -- 172.21.15.102:0/270948129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cf4108360 msgr2=0x7f3cf419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cf4108360 0x7f3cf419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf3fff700 1 -- 172.21.15.102:0/270948129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf419f290 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf37fe700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3cf4102cb0 0x7f3cf419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.754+0000 7f3cf8d90700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cf4108360 0x7f3cf419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.755+0000 7f3cf3fff700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf419a280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00e9a0 tx=0x7f3cdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.755+0000 7f3cf17fa700 1 -- 172.21.15.102:0/270948129 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00ca10 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.755+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf419f580 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.755+0000 7f3cf17fa700 1 -- 172.21.15.102:0/270948129 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f3cdc00cbb0 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.755+0000 7f3cf17fa700 1 -- 172.21.15.102:0/270948129 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc018680 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.755+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf41a6e00 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.756+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf4068f00 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.756+0000 7f3cf17fa700 1 -- 172.21.15.102:0/270948129 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f3cdc010430 con 0x7f3cf41022f0 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.757+0000 7f3cf17fa700 1 --2- 172.21.15.102:0/270948129 >> v2:172.21.15.102:6800/622906077 conn(0x7f3ce006dc00 0x7f3ce00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:52.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.757+0000 7f3cf37fe700 1 --2- 172.21.15.102:0/270948129 >> v2:172.21.15.102:6800/622906077 conn(0x7f3ce006dc00 0x7f3ce00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.757+0000 7f3cf17fa700 1 -- 172.21.15.102:0/270948129 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f3cdc08dcf0 con 0x7f3cf41022f0 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.757+0000 7f3cf37fe700 1 --2- 172.21.15.102:0/270948129 >> v2:172.21.15.102:6800/622906077 conn(0x7f3ce006dc00 0x7f3ce00700c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400b2e0 tx=0x7f3ce401a040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.761+0000 7f3cf17fa700 1 -- 172.21.15.102:0/270948129 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3cdc092070 con 0x7f3cf41022f0 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.994+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3cf41a6fd0 con 0x7f3cf41022f0 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stdout:128849018913 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.995+0000 7f3cf17fa700 1 -- 172.21.15.102:0/270948129 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3cdc05b7d0 con 0x7f3cf41022f0 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 >> v2:172.21.15.102:6800/622906077 conn(0x7f3ce006dc00 msgr2=0x7f3ce00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/270948129 >> v2:172.21.15.102:6800/622906077 conn(0x7f3ce006dc00 0x7f3ce00700c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400b2e0 tx=0x7f3ce401a040 comp rx=0 tx=0).stop 2024-03-30T13:57:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 msgr2=0x7f3cf419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf419a280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00e9a0 tx=0x7f3cdc00ecb0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 shutdown_connections 2024-03-30T13:57:52.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf41022f0 0x7f3cf419a280 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/270948129 >> v2:172.21.15.102:6800/622906077 conn(0x7f3ce006dc00 0x7f3ce00700c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3cf4108360 0x7f3cf419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 --2- 172.21.15.102:0/270948129 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3cf4102cb0 0x7f3cf419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 >> 172.21.15.102:0/270948129 conn(0x7f3cf4079310 msgr2=0x7f3cf4074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 shutdown_connections 2024-03-30T13:57:52.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:51.997+0000 7f3cfa7f3700 1 -- 172.21.15.102:0/270948129 wait complete. 2024-03-30T13:57:53.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:53 smithi102 ceph-mon[29392]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:53.334 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:53 smithi102 ceph-mon[37717]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:53.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:53 smithi138 ceph-mon[33397]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:54.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182436 got 107374182439 for osd.3 2024-03-30T13:57:54.514 DEBUG:teuthology.parallel:result is None 2024-03-30T13:57:54.957 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018913 for osd.4 2024-03-30T13:57:54.957 DEBUG:teuthology.parallel:result is None 2024-03-30T13:57:55.444 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495637 got 197568495639 for osd.6 2024-03-30T13:57:55.445 DEBUG:teuthology.parallel:result is None 2024-03-30T13:57:55.558 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:55 smithi102 ceph-mon[29392]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:55.558 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:55 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:55.559 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:55 smithi102 ceph-mon[37717]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:55.559 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:55 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:55.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:55 smithi138 ceph-mon[33397]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:55.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:55 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:57:55.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:57:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:57:55.988 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234000 got 231928234001 for osd.7 2024-03-30T13:57:55.988 DEBUG:teuthology.parallel:result is None 2024-03-30T13:57:56.509 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:56.510 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:56.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:57:56.739 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:57:56.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 -- 172.21.15.102:0/1274930608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928106900 msgr2=0x7f0928106ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:56.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 --2- 172.21.15.102:0/1274930608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928106900 0x7f0928106ce0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0918009990 tx=0x7f0918009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 -- 172.21.15.102:0/1274930608 shutdown_connections 2024-03-30T13:57:56.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 --2- 172.21.15.102:0/1274930608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928106900 0x7f0928106ce0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 --2- 172.21.15.102:0/1274930608 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 0x7f092810d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 --2- 172.21.15.102:0/1274930608 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928100900 0x7f0928100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 -- 172.21.15.102:0/1274930608 >> 172.21.15.102:0/1274930608 conn(0x7f0928077740 msgr2=0x7f09280feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 -- 172.21.15.102:0/1274930608 shutdown_connections 2024-03-30T13:57:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.755+0000 7f092e721700 1 -- 172.21.15.102:0/1274930608 wait complete. 2024-03-30T13:57:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 Processor -- start 2024-03-30T13:57:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 -- start start 2024-03-30T13:57:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928100900 0x7f09281980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 0x7f0928198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928106900 0x7f092819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0928110c10 con 0x7f0928100900 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0928110a90 con 0x7f0928106900 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.756+0000 7f092e721700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0928110d90 con 0x7f09281012c0 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 0x7f0928198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f0927fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928100900 0x7f09281980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 0x7f0928198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:35094/0 (socket says 172.21.15.102:35094) 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 -- 172.21.15.102:0/597207440 learned_addr learned my addr 172.21.15.102:0/597207440 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f092ccbe700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928106900 0x7f092819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 -- 172.21.15.102:0/597207440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928106900 msgr2=0x7f092819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928106900 0x7f092819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 -- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928100900 msgr2=0x7f09281980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928100900 0x7f09281980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 -- 172.21.15.102:0/597207440 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092819d100 con 0x7f09281012c0 2024-03-30T13:57:56.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f0927fff700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928100900 0x7f09281980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:56.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09277fe700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 0x7f0928198630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f091c00d980 tx=0x7f091c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:56.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09257fa700 1 -- 172.21.15.102:0/597207440 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c016070 con 0x7f09281012c0 2024-03-30T13:57:56.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092819d3f0 con 0x7f09281012c0 2024-03-30T13:57:56.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09257fa700 1 -- 172.21.15.102:0/597207440 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f091c004830 con 0x7f09281012c0 2024-03-30T13:57:56.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.757+0000 7f09257fa700 1 -- 172.21.15.102:0/597207440 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c005510 con 0x7f09281012c0 2024-03-30T13:57:56.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.758+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09281a4c70 con 0x7f09281012c0 2024-03-30T13:57:56.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.759+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0928102040 con 0x7f09281012c0 2024-03-30T13:57:56.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.759+0000 7f09257fa700 1 -- 172.21.15.102:0/597207440 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f091c00f690 con 0x7f09281012c0 2024-03-30T13:57:56.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.760+0000 7f09257fa700 1 --2- 172.21.15.102:0/597207440 >> v2:172.21.15.102:6800/622906077 conn(0x7f091006dc00 0x7f09100700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:56.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.760+0000 7f09257fa700 1 -- 172.21.15.102:0/597207440 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f091c08d8c0 con 0x7f09281012c0 2024-03-30T13:57:56.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.760+0000 7f0927fff700 1 --2- 172.21.15.102:0/597207440 >> v2:172.21.15.102:6800/622906077 conn(0x7f091006dc00 0x7f09100700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:56.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.760+0000 7f0927fff700 1 --2- 172.21.15.102:0/597207440 >> v2:172.21.15.102:6800/622906077 conn(0x7f091006dc00 0x7f09100700c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09180098e0 tx=0x7f0918005960 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.768+0000 7f09257fa700 1 -- 172.21.15.102:0/597207440 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f091c01f020 con 0x7f09281012c0 2024-03-30T13:57:56.764 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:57:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:57:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:57:56.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.970+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0928199450 con 0x7f09281012c0 2024-03-30T13:57:56.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.971+0000 7f09257fa700 1 -- 172.21.15.102:0/597207440 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f091c063600 con 0x7f09281012c0 2024-03-30T13:57:56.967 INFO:teuthology.orchestra.run.smithi102.stdout:55834574898 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 >> v2:172.21.15.102:6800/622906077 conn(0x7f091006dc00 msgr2=0x7f09100700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 --2- 172.21.15.102:0/597207440 >> v2:172.21.15.102:6800/622906077 conn(0x7f091006dc00 0x7f09100700c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f09180098e0 tx=0x7f0918005960 comp rx=0 tx=0).stop 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 msgr2=0x7f0928198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 0x7f0928198630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f091c00d980 tx=0x7f091c00dc90 comp rx=0 tx=0).stop 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 shutdown_connections 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0928100900 0x7f09281980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 --2- 172.21.15.102:0/597207440 >> v2:172.21.15.102:6800/622906077 conn(0x7f091006dc00 0x7f09100700c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928106900 0x7f092819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 --2- 172.21.15.102:0/597207440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09281012c0 0x7f0928198630 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 >> 172.21.15.102:0/597207440 conn(0x7f0928077740 msgr2=0x7f092810e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 shutdown_connections 2024-03-30T13:57:56.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:56.973+0000 7f092e721700 1 -- 172.21.15.102:0/597207440 wait complete. 2024-03-30T13:57:57.594 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:57 smithi102 ceph-mon[29392]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:57.594 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/597207440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T13:57:57.594 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:57 smithi102 ceph-mon[37717]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:57.594 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/597207440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T13:57:57.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:57 smithi138 ceph-mon[33397]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:57:57.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/597207440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T13:57:59.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:57:59 smithi138 ceph-mon[33397]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:59.599 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:57:59 smithi102 ceph-mon[37717]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:59.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:57:59 smithi102 ceph-mon[29392]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:57:59.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/210529807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 msgr2=0x7fbe08108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/210529807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 0x7fbe08108650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009990 tx=0x7fbdf8009ca0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/210529807 shutdown_connections 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/210529807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 0x7fbe0810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/210529807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 0x7fbe08108650 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/210529807 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe08102290 0x7fbe08102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/210529807 >> 172.21.15.102:0/210529807 conn(0x7fbe08079370 msgr2=0x7fbe08079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/210529807 shutdown_connections 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.603+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/210529807 wait complete. 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 Processor -- start 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 -- start start 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe08102290 0x7fbe08198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 0x7fbe08198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 0x7fbe0819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbe08111bc0 con 0x7fbe08102c50 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe08111a40 con 0x7fbe08102290 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.604+0000 7fbe0d4e5700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbe08111d40 con 0x7fbe08108270 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 0x7fbe08198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 0x7fbe08198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47144/0 (socket says 172.21.15.102:47144) 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 -- 172.21.15.102:0/3600897586 learned_addr learned my addr 172.21.15.102:0/3600897586 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe077fe700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 0x7fbe0819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe06ffd700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe08102290 0x7fbe08198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 -- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 msgr2=0x7fbe0819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 0x7fbe0819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 -- 172.21.15.102:0/3600897586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe08102290 msgr2=0x7fbe08198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe08102290 0x7fbe08198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 -- 172.21.15.102:0/3600897586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0819d050 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe067fc700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 0x7fbe08198580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00d980 tx=0x7fbdfc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbdeffff700 1 -- 172.21.15.102:0/3600897586 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc017070 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0819d340 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbdeffff700 1 -- 172.21.15.102:0/3600897586 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fbdfc005020 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbdeffff700 1 -- 172.21.15.102:0/3600897586 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc009970 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.605+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe081a4bc0 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.606+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe08068f00 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.607+0000 7fbdeffff700 1 -- 172.21.15.102:0/3600897586 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fbdfc009b10 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.607+0000 7fbdeffff700 1 --2- 172.21.15.102:0/3600897586 >> v2:172.21.15.102:6800/622906077 conn(0x7fbdf006dc00 0x7fbdf00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.607+0000 7fbe06ffd700 1 --2- 172.21.15.102:0/3600897586 >> v2:172.21.15.102:6800/622906077 conn(0x7fbdf006dc00 0x7fbdf00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.607+0000 7fbdeffff700 1 -- 172.21.15.102:0/3600897586 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7fbdfc08d9c0 con 0x7fbe08102c50 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.608+0000 7fbe06ffd700 1 --2- 172.21.15.102:0/3600897586 >> v2:172.21.15.102:6800/622906077 conn(0x7fbdf006dc00 0x7fbdf00700c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800b2e0 tx=0x7fbdf8005960 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:59.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.612+0000 7fbdeffff700 1 -- 172.21.15.102:0/3600897586 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbdfc08fe20 con 0x7fbe08102c50 2024-03-30T13:57:59.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.811+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbe081a4df0 con 0x7fbe08102c50 2024-03-30T13:57:59.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.812+0000 7fbdeffff700 1 -- 172.21.15.102:0/3600897586 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbdfc020020 con 0x7fbe08102c50 2024-03-30T13:57:59.807 INFO:teuthology.orchestra.run.smithi102.stdout:163208757276 2024-03-30T13:57:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.813+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 >> v2:172.21.15.102:6800/622906077 conn(0x7fbdf006dc00 msgr2=0x7fbdf00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/3600897586 >> v2:172.21.15.102:6800/622906077 conn(0x7fbdf006dc00 0x7fbdf00700c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800b2e0 tx=0x7fbdf8005960 comp rx=0 tx=0).stop 2024-03-30T13:57:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 msgr2=0x7fbe08198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 0x7fbe08198580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00d980 tx=0x7fbdfc00dc90 comp rx=0 tx=0).stop 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 shutdown_connections 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/3600897586 >> v2:172.21.15.102:6800/622906077 conn(0x7fbdf006dc00 0x7fbdf00700c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe08108270 0x7fbe0819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe08102290 0x7fbe08198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 --2- 172.21.15.102:0/3600897586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe08102c50 0x7fbe08198580 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 >> 172.21.15.102:0/3600897586 conn(0x7fbe08079370 msgr2=0x7fbe08105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 shutdown_connections 2024-03-30T13:57:59.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.814+0000 7fbe0d4e5700 1 -- 172.21.15.102:0/3600897586 wait complete. 2024-03-30T13:57:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 -- 172.21.15.102:0/1207081605 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406b390 msgr2=0x7f08e4106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1207081605 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406b390 0x7f08e4106e90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009920 tx=0x7f08d8009c30 comp rx=0 tx=0).stop 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 -- 172.21.15.102:0/1207081605 shutdown_connections 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1207081605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e41075c0 0x7f08e410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1207081605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08e406a9e0 0x7f08e406adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1207081605 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406b390 0x7f08e4106e90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 -- 172.21.15.102:0/1207081605 >> 172.21.15.102:0/1207081605 conn(0x7f08e4077190 msgr2=0x7f08e40775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 -- 172.21.15.102:0/1207081605 shutdown_connections 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.933+0000 7f08ea7be700 1 -- 172.21.15.102:0/1207081605 wait complete. 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 Processor -- start 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 -- start start 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406a9e0 0x7f08e4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e406b390 0x7f08e4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08e41075c0 0x7f08e419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4111c30 con 0x7f08e406b390 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4111ab0 con 0x7f08e41075c0 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08ea7be700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f08e4111db0 con 0x7f08e406a9e0 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406a9e0 0x7f08e4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406a9e0 0x7f08e4198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:35126/0 (socket says 172.21.15.102:35126) 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 -- 172.21.15.102:0/1783042459 learned_addr learned my addr 172.21.15.102:0/1783042459 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e37fe700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e406b390 0x7f08e4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e8d5b700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08e41075c0 0x7f08e419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 -- 172.21.15.102:0/1783042459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08e41075c0 msgr2=0x7f08e419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08e41075c0 0x7f08e419cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 -- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e406b390 msgr2=0x7f08e4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e406b390 0x7f08e4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e3fff700 1 -- 172.21.15.102:0/1783042459 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e419d240 con 0x7f08e406a9e0 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e37fe700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e406b390 0x7f08e4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:57:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.934+0000 7f08e8d5b700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08e41075c0 0x7f08e419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.935+0000 7f08e3fff700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406a9e0 0x7f08e4198230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f08d400d980 tx=0x7f08d400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.935+0000 7f08e17fa700 1 -- 172.21.15.102:0/1783042459 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4017070 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.935+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e419d530 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.935+0000 7f08e17fa700 1 -- 172.21.15.102:0/1783042459 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f08d4005020 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.935+0000 7f08e17fa700 1 -- 172.21.15.102:0/1783042459 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4009b20 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.935+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e41a4db0 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.936+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e4103970 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.936+0000 7f08e17fa700 1 -- 172.21.15.102:0/1783042459 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f08d4009cc0 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.937+0000 7f08e17fa700 1 --2- 172.21.15.102:0/1783042459 >> v2:172.21.15.102:6800/622906077 conn(0x7f08cc06dc00 0x7f08cc0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.937+0000 7f08e17fa700 1 -- 172.21.15.102:0/1783042459 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f08d408da40 con 0x7f08e406a9e0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.937+0000 7f08e37fe700 1 --2- 172.21.15.102:0/1783042459 >> v2:172.21.15.102:6800/622906077 conn(0x7f08cc06dc00 0x7f08cc0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.937+0000 7f08e37fe700 1 --2- 172.21.15.102:0/1783042459 >> v2:172.21.15.102:6800/622906077 conn(0x7f08cc06dc00 0x7f08cc0700c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f08d800c010 tx=0x7f08d8019040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:57:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:57:59.941+0000 7f08e17fa700 1 -- 172.21.15.102:0/1783042459 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f08d4020020 con 0x7f08e406a9e0 2024-03-30T13:58:00.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.165+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f08e4199590 con 0x7f08e406a9e0 2024-03-30T13:58:00.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.165+0000 7f08e17fa700 1 -- 172.21.15.102:0/1783042459 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f08d40637a0 con 0x7f08e406a9e0 2024-03-30T13:58:00.161 INFO:teuthology.orchestra.run.smithi102.stdout:34359738423 2024-03-30T13:58:00.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 >> v2:172.21.15.102:6800/622906077 conn(0x7f08cc06dc00 msgr2=0x7f08cc0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:00.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1783042459 >> v2:172.21.15.102:6800/622906077 conn(0x7f08cc06dc00 0x7f08cc0700c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f08d800c010 tx=0x7f08d8019040 comp rx=0 tx=0).stop 2024-03-30T13:58:00.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406a9e0 msgr2=0x7f08e4198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406a9e0 0x7f08e4198230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f08d400d980 tx=0x7f08d400dc90 comp rx=0 tx=0).stop 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 shutdown_connections 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1783042459 >> v2:172.21.15.102:6800/622906077 conn(0x7f08cc06dc00 0x7f08cc0700c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08e41075c0 0x7f08e419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f08e406a9e0 0x7f08e4198230 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 --2- 172.21.15.102:0/1783042459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e406b390 0x7f08e4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 >> 172.21.15.102:0/1783042459 conn(0x7f08e4077190 msgr2=0x7f08e40fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 shutdown_connections 2024-03-30T13:58:00.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:00.168+0000 7f08ea7be700 1 -- 172.21.15.102:0/1783042459 wait complete. 2024-03-30T13:58:00.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3600897586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T13:58:00.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1783042459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T13:58:00.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3600897586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T13:58:00.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1783042459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T13:58:00.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3600897586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T13:58:00.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1783042459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T13:58:01.597 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:01 smithi102 ceph-mon[37717]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:01.597 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:01 smithi102 ceph-mon[29392]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:01.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574898 for osd.1 2024-03-30T13:58:01.623 DEBUG:teuthology.parallel:result is None 2024-03-30T13:58:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:01 smithi138 ceph-mon[33397]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/4189129332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df810ecf0 msgr2=0x7f2df81110e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/4189129332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df810ecf0 0x7f2df81110e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009f90 tx=0x7f2df000b330 comp rx=0 tx=0).stop 2024-03-30T13:58:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/4189129332 shutdown_connections 2024-03-30T13:58:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/4189129332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df810ecf0 0x7f2df81110e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/4189129332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2df8069ef0 0x7f2df806a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/4189129332 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 0x7f2df810e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/4189129332 >> 172.21.15.102:0/4189129332 conn(0x7f2df8078070 msgr2=0x7f2df8078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.403+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/4189129332 shutdown_connections 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.404+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/4189129332 wait complete. 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.404+0000 7f2dfcdb6700 1 Processor -- start 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.404+0000 7f2dfcdb6700 1 -- start start 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.404+0000 7f2dfcdb6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df8069ef0 0x7f2df819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.404+0000 7f2dfcdb6700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 0x7f2df819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2dfcdb6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2df810ecf0 0x7f2df819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2dfcdb6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8113e10 con 0x7f2df8069ef0 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2dfcdb6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8113c90 con 0x7f2df810ecf0 2024-03-30T13:58:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2dfcdb6700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2df8113f90 con 0x7f2df806a810 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df8069ef0 0x7f2df819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df5d9b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 0x7f2df819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df8069ef0 0x7f2df819a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42304/0 (socket says 172.21.15.102:42304) 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 -- 172.21.15.102:0/2290229246 learned_addr learned my addr 172.21.15.102:0/2290229246 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df5d9b700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 0x7f2df819a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:45540/0 (socket says 172.21.15.102:45540) 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df6d9d700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2df810ecf0 0x7f2df819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 -- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 msgr2=0x7f2df819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 0x7f2df819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 -- 172.21.15.102:0/2290229246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2df810ecf0 msgr2=0x7f2df819eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2df810ecf0 0x7f2df819eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 -- 172.21.15.102:0/2290229246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df819f3e0 con 0x7f2df8069ef0 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df5d9b700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 0x7f2df819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:58:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df6d9d700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2df810ecf0 0x7f2df819eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2df659c700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df8069ef0 0x7f2df819a3d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00b530 tx=0x7f2dec00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2de77fe700 1 -- 172.21.15.102:0/2290229246 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec015020 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df819f6d0 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2de77fe700 1 -- 172.21.15.102:0/2290229246 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f2dec011ce0 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.405+0000 7f2de77fe700 1 -- 172.21.15.102:0/2290229246 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec010e10 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.406+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df81a6f50 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.406+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df8068f00 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.407+0000 7f2de77fe700 1 -- 172.21.15.102:0/2290229246 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f2dec018020 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.407+0000 7f2de77fe700 1 --2- 172.21.15.102:0/2290229246 >> v2:172.21.15.102:6800/622906077 conn(0x7f2de006dc00 0x7f2de00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.407+0000 7f2df5d9b700 1 --2- 172.21.15.102:0/2290229246 >> v2:172.21.15.102:6800/622906077 conn(0x7f2de006dc00 0x7f2de00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.407+0000 7f2de77fe700 1 -- 172.21.15.102:0/2290229246 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f2dec08d410 con 0x7f2df8069ef0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.408+0000 7f2df5d9b700 1 --2- 172.21.15.102:0/2290229246 >> v2:172.21.15.102:6800/622906077 conn(0x7f2de006dc00 0x7f2de00700c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009750 tx=0x7f2de8009310 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.412+0000 7f2de77fe700 1 -- 172.21.15.102:0/2290229246 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2dec01f020 con 0x7f2df8069ef0 2024-03-30T13:58:02.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.610+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f2df81a7120 con 0x7f2df8069ef0 2024-03-30T13:58:02.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.611+0000 7f2de77fe700 1 -- 172.21.15.102:0/2290229246 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2dec05aef0 con 0x7f2df8069ef0 2024-03-30T13:58:02.606 INFO:teuthology.orchestra.run.smithi102.stdout:77309411374 2024-03-30T13:58:02.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 >> v2:172.21.15.102:6800/622906077 conn(0x7f2de006dc00 msgr2=0x7f2de00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/2290229246 >> v2:172.21.15.102:6800/622906077 conn(0x7f2de006dc00 0x7f2de00700c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009750 tx=0x7f2de8009310 comp rx=0 tx=0).stop 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df8069ef0 msgr2=0x7f2df819a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df8069ef0 0x7f2df819a3d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00b530 tx=0x7f2dec00b840 comp rx=0 tx=0).stop 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 shutdown_connections 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df8069ef0 0x7f2df819a3d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/2290229246 >> v2:172.21.15.102:6800/622906077 conn(0x7f2de006dc00 0x7f2de00700c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2df810ecf0 0x7f2df819eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 --2- 172.21.15.102:0/2290229246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2df806a810 0x7f2df819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 >> 172.21.15.102:0/2290229246 conn(0x7f2df8078070 msgr2=0x7f2df8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 shutdown_connections 2024-03-30T13:58:02.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:02.613+0000 7f2dfcdb6700 1 -- 172.21.15.102:0/2290229246 wait complete. 2024-03-30T13:58:03.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:03 smithi138 ceph-mon[33397]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:03.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2290229246' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T13:58:03.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:03 smithi102 ceph-mon[29392]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:03.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2290229246' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T13:58:03.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:03 smithi102 ceph-mon[37717]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:03.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2290229246' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T13:58:03.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411374 for osd.2 2024-03-30T13:58:03.779 DEBUG:teuthology.parallel:result is None 2024-03-30T13:58:04.331 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757276 for osd.5 2024-03-30T13:58:04.331 DEBUG:teuthology.parallel:result is None 2024-03-30T13:58:04.341 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738423 for osd.0 2024-03-30T13:58:04.341 DEBUG:teuthology.parallel:result is None 2024-03-30T13:58:04.341 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T13:58:04.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:58:05.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:05 smithi138 ceph-mon[33397]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:05.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:58:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:58:05.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:05 smithi102 ceph-mon[29392]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:05.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:05 smithi102 ceph-mon[37717]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:06.460 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:06.586 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:06.586 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:06.587 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:58:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:06.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:07.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:07 smithi138 ceph-mon[33397]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:07.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:07 smithi102 ceph-mon[29392]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:07.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:07 smithi102 ceph-mon[37717]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:08.440 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 podman[72337]: 2024-03-30 13:58:08.309394905 +0000 UTC m=+0.477429445 container create ff4acdb16fe1ec53ba463b4fcf0f7fa6ded500e6629e0232425a66587aac6b90 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 -- 172.21.15.102:0/2310919989 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1330101d80 msgr2=0x7f1330105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 --2- 172.21.15.102:0/2310919989 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1330101d80 0x7f1330105d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f132c009920 tx=0x7f132c009c30 comp rx=0 tx=0).stop 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 -- 172.21.15.102:0/2310919989 shutdown_connections 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 --2- 172.21.15.102:0/2310919989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13301013d0 0x7f13301017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 --2- 172.21.15.102:0/2310919989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1330106420 0x7f133010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 --2- 172.21.15.102:0/2310919989 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1330101d80 0x7f1330105d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 -- 172.21.15.102:0/2310919989 >> 172.21.15.102:0/2310919989 conn(0x7f1330077190 msgr2=0x7f13300775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 -- 172.21.15.102:0/2310919989 shutdown_connections 2024-03-30T13:58:08.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.740+0000 7f133823c700 1 -- 172.21.15.102:0/2310919989 wait complete. 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 Processor -- start 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 -- start start 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f13301013d0 0x7f1330196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1330101d80 0x7f13301972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1330106420 0x7f1330197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1330110af0 con 0x7f1330106420 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1330110970 con 0x7f1330101d80 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f133823c700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1330110c70 con 0x7f13301013d0 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f1335fd8700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f13301013d0 0x7f1330196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f1335fd8700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f13301013d0 0x7f1330196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:45554/0 (socket says 172.21.15.102:45554) 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f1335fd8700 1 -- 172.21.15.102:0/2790084685 learned_addr learned my addr 172.21.15.102:0/2790084685 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f13367d9700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1330106420 0x7f1330197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.741+0000 7f13357d7700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1330101d80 0x7f13301972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1335fd8700 1 -- 172.21.15.102:0/2790084685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1330101d80 msgr2=0x7f13301972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1335fd8700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1330101d80 0x7f13301972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1335fd8700 1 -- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1330106420 msgr2=0x7f1330197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1335fd8700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1330106420 0x7f1330197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1335fd8700 1 -- 172.21.15.102:0/2790084685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13301a6600 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1335fd8700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f13301013d0 0x7f1330196d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f132000ec10 tx=0x7f132000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1326ffd700 1 -- 172.21.15.102:0/2790084685 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132000cc90 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13301a68f0 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1326ffd700 1 -- 172.21.15.102:0/2790084685 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f132000ce30 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f1326ffd700 1 -- 172.21.15.102:0/2790084685 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320010430 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.742+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13301a6e50 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.743+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1330068f00 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.744+0000 7f1326ffd700 1 -- 172.21.15.102:0/2790084685 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f1320004020 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.744+0000 7f1326ffd700 1 --2- 172.21.15.102:0/2790084685 >> v2:172.21.15.102:6800/622906077 conn(0x7f131c06dc00 0x7f131c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.744+0000 7f1326ffd700 1 -- 172.21.15.102:0/2790084685 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f132008de00 con 0x7f13301013d0 2024-03-30T13:58:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.744+0000 7f13357d7700 1 --2- 172.21.15.102:0/2790084685 >> v2:172.21.15.102:6800/622906077 conn(0x7f131c06dc00 0x7f131c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:08.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.745+0000 7f13357d7700 1 --2- 172.21.15.102:0/2790084685 >> v2:172.21.15.102:6800/622906077 conn(0x7f131c06dc00 0x7f131c0700c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f132c0098f0 tx=0x7f132c019040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:08.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.751+0000 7f1326ffd700 1 -- 172.21.15.102:0/2790084685 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f132005b8e0 con 0x7f13301013d0 2024-03-30T13:58:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.952+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 --> v2:172.21.15.102:6800/622906077 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f13300ff6f0 con 0x7f131c06dc00 2024-03-30T13:58:08.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.963+0000 7f1326ffd700 1 -- 172.21.15.102:0/2790084685 <== mgr.24464 v2:172.21.15.102:6800/622906077 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346464 (secure 0 0 0) 0x7f13300ff6f0 con 0x7f131c06dc00 2024-03-30T13:58:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 >> v2:172.21.15.102:6800/622906077 conn(0x7f131c06dc00 msgr2=0x7f131c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 --2- 172.21.15.102:0/2790084685 >> v2:172.21.15.102:6800/622906077 conn(0x7f131c06dc00 0x7f131c0700c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f132c0098f0 tx=0x7f132c019040 comp rx=0 tx=0).stop 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f13301013d0 msgr2=0x7f1330196d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f13301013d0 0x7f1330196d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f132000ec10 tx=0x7f132000ef20 comp rx=0 tx=0).stop 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 shutdown_connections 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1330106420 0x7f1330197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 --2- 172.21.15.102:0/2790084685 >> v2:172.21.15.102:6800/622906077 conn(0x7f131c06dc00 0x7f131c0700c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f13301013d0 0x7f1330196d90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 --2- 172.21.15.102:0/2790084685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1330101d80 0x7f13301972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.966+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 >> 172.21.15.102:0/2790084685 conn(0x7f1330077190 msgr2=0x7f13300fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.967+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 shutdown_connections 2024-03-30T13:58:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:08.967+0000 7f133823c700 1 -- 172.21.15.102:0/2790084685 wait complete. 2024-03-30T13:58:08.968 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-30T13:58:09.185 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 podman[72337]: 2024-03-30 13:58:08.880561552 +0000 UTC m=+1.048596091 container init ff4acdb16fe1ec53ba463b4fcf0f7fa6ded500e6629e0232425a66587aac6b90 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:09.185 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 podman[72337]: 2024-03-30 13:58:08.883520287 +0000 UTC m=+1.051554833 container start ff4acdb16fe1ec53ba463b4fcf0f7fa6ded500e6629e0232425a66587aac6b90 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:09.185 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.895Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-30T13:58:09.185 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.895Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-30T13:58:09.185 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.897Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-03-30T13:58:09.185 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.897Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-30T13:58:09.186 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.925Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:58:09.186 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.925Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:58:09.186 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.929Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-30T13:58:09.186 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:08.929Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-30T13:58:09.186 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 bash[72337]: ff4acdb16fe1ec53ba463b4fcf0f7fa6ded500e6629e0232425a66587aac6b90 2024-03-30T13:58:09.186 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:08 smithi102 systemd[1]: Started Ceph alertmanager.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:09.438 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:09 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:09] ENGINE Bus STOPPING 2024-03-30T13:58:09.438 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:09 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:58:09.438 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:09 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:09] ENGINE Bus STOPPED 2024-03-30T13:58:09.438 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:09 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:09] ENGINE Bus STARTING 2024-03-30T13:58:09.438 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:09 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:09] ENGINE Serving on http://:::9283 2024-03-30T13:58:09.439 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:09 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:09] ENGINE Bus STARTED 2024-03-30T13:58:09.700 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-30T13:58:08.233290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1007,"num_read_kb":853,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240180,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721036,"statfs":{"total":767960285184,"available":767714340864,"internally_reserved":0,"allocated":7565312,"data_stored":4027914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003399"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203242+0000","last_change":"2024-03-30T13:56:51.145179+0000","last_active":"2024-03-30T13:57:24.203242+0000","last_peered":"2024-03-30T13:57:24.203242+0000","last_clean":"2024-03-30T13:57:24.203242+0000","last_became_active":"2024-03-30T13:56:51.144878+0000","last_became_peered":"2024-03-30T13:56:51.144878+0000","last_unstale":"2024-03-30T13:57:24.203242+0000","last_undegraded":"2024-03-30T13:57:24.203242+0000","last_fullsized":"2024-03-30T13:57:24.203242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:34:38.183139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204856+0000","last_change":"2024-03-30T13:56:49.138594+0000","last_active":"2024-03-30T13:57:24.204856+0000","last_peered":"2024-03-30T13:57:24.204856+0000","last_clean":"2024-03-30T13:57:24.204856+0000","last_became_active":"2024-03-30T13:56:49.138275+0000","last_became_peered":"2024-03-30T13:56:49.138275+0000","last_unstale":"2024-03-30T13:57:24.204856+0000","last_undegraded":"2024-03-30T13:57:24.204856+0000","last_fullsized":"2024-03-30T13:57:24.204856+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:15:00.924078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202536+0000","last_change":"2024-03-30T13:56:45.111202+0000","last_active":"2024-03-30T13:57:24.202536+0000","last_peered":"2024-03-30T13:57:24.202536+0000","last_clean":"2024-03-30T13:57:24.202536+0000","last_became_active":"2024-03-30T13:56:45.110871+0000","last_became_peered":"2024-03-30T13:56:45.110871+0000","last_unstale":"2024-03-30T13:57:24.202536+0000","last_undegraded":"2024-03-30T13:57:24.202536+0000","last_fullsized":"2024-03-30T13:57:24.202536+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:11:00.993985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203097+0000","last_change":"2024-03-30T13:56:47.122055+0000","last_active":"2024-03-30T13:57:24.203097+0000","last_peered":"2024-03-30T13:57:24.203097+0000","last_clean":"2024-03-30T13:57:24.203097+0000","last_became_active":"2024-03-30T13:56:47.121290+0000","last_became_peered":"2024-03-30T13:56:47.121290+0000","last_unstale":"2024-03-30T13:57:24.203097+0000","last_undegraded":"2024-03-30T13:57:24.203097+0000","last_fullsized":"2024-03-30T13:57:24.203097+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:57:40.768021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612707+0000","last_change":"2024-03-30T13:56:51.377451+0000","last_active":"2024-03-30T13:57:24.612707+0000","last_peered":"2024-03-30T13:57:24.612707+0000","last_clean":"2024-03-30T13:57:24.612707+0000","last_became_active":"2024-03-30T13:56:51.376678+0000","last_became_peered":"2024-03-30T13:56:51.376678+0000","last_unstale":"2024-03-30T13:57:24.612707+0000","last_undegraded":"2024-03-30T13:57:24.612707+0000","last_fullsized":"2024-03-30T13:57:24.612707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:46:01.897415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203097+0000","last_change":"2024-03-30T13:56:49.137668+0000","last_active":"2024-03-30T13:57:24.203097+0000","last_peered":"2024-03-30T13:57:24.203097+0000","last_clean":"2024-03-30T13:57:24.203097+0000","last_became_active":"2024-03-30T13:56:49.137322+0000","last_became_peered":"2024-03-30T13:56:49.137322+0000","last_unstale":"2024-03-30T13:57:24.203097+0000","last_undegraded":"2024-03-30T13:57:24.203097+0000","last_fullsized":"2024-03-30T13:57:24.203097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:46:03.272319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613263+0000","last_change":"2024-03-30T13:56:45.121812+0000","last_active":"2024-03-30T13:57:24.613263+0000","last_peered":"2024-03-30T13:57:24.613263+0000","last_clean":"2024-03-30T13:57:24.613263+0000","last_became_active":"2024-03-30T13:56:45.121401+0000","last_became_peered":"2024-03-30T13:56:45.121401+0000","last_unstale":"2024-03-30T13:57:24.613263+0000","last_undegraded":"2024-03-30T13:57:24.613263+0000","last_fullsized":"2024-03-30T13:57:24.613263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:55:32.205894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612606+0000","last_change":"2024-03-30T13:56:47.129602+0000","last_active":"2024-03-30T13:57:24.612606+0000","last_peered":"2024-03-30T13:57:24.612606+0000","last_clean":"2024-03-30T13:57:24.612606+0000","last_became_active":"2024-03-30T13:56:47.129289+0000","last_became_peered":"2024-03-30T13:56:47.129289+0000","last_unstale":"2024-03-30T13:57:24.612606+0000","last_undegraded":"2024-03-30T13:57:24.612606+0000","last_fullsized":"2024-03-30T13:57:24.612606+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:27:19.864234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202206+0000","last_change":"2024-03-30T13:56:51.147121+0000","last_active":"2024-03-30T13:57:24.202206+0000","last_peered":"2024-03-30T13:57:24.202206+0000","last_clean":"2024-03-30T13:57:24.202206+0000","last_became_active":"2024-03-30T13:56:51.145364+0000","last_became_peered":"2024-03-30T13:56:51.145364+0000","last_unstale":"2024-03-30T13:57:24.202206+0000","last_undegraded":"2024-03-30T13:57:24.202206+0000","last_fullsized":"2024-03-30T13:57:24.202206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:41:10.749407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291339+0000","last_change":"2024-03-30T13:56:49.141397+0000","last_active":"2024-03-30T13:57:24.291339+0000","last_peered":"2024-03-30T13:57:24.291339+0000","last_clean":"2024-03-30T13:57:24.291339+0000","last_became_active":"2024-03-30T13:56:49.141111+0000","last_became_peered":"2024-03-30T13:56:49.141111+0000","last_unstale":"2024-03-30T13:57:24.291339+0000","last_undegraded":"2024-03-30T13:57:24.291339+0000","last_fullsized":"2024-03-30T13:57:24.291339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:16:27.094352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291434+0000","last_change":"2024-03-30T13:56:45.121753+0000","last_active":"2024-03-30T13:57:24.291434+0000","last_peered":"2024-03-30T13:57:24.291434+0000","last_clean":"2024-03-30T13:57:24.291434+0000","last_became_active":"2024-03-30T13:56:45.120978+0000","last_became_peered":"2024-03-30T13:56:45.120978+0000","last_unstale":"2024-03-30T13:57:24.291434+0000","last_undegraded":"2024-03-30T13:57:24.291434+0000","last_fullsized":"2024-03-30T13:57:24.291434+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:54:43.154356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203716+0000","last_change":"2024-03-30T13:56:47.133118+0000","last_active":"2024-03-30T13:57:24.203716+0000","last_peered":"2024-03-30T13:57:24.203716+0000","last_clean":"2024-03-30T13:57:24.203716+0000","last_became_active":"2024-03-30T13:56:47.132851+0000","last_became_peered":"2024-03-30T13:56:47.132851+0000","last_unstale":"2024-03-30T13:57:24.203716+0000","last_undegraded":"2024-03-30T13:57:24.203716+0000","last_fullsized":"2024-03-30T13:57:24.203716+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:31:33.961265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291069+0000","last_change":"2024-03-30T13:56:51.145874+0000","last_active":"2024-03-30T13:57:24.291069+0000","last_peered":"2024-03-30T13:57:24.291069+0000","last_clean":"2024-03-30T13:57:24.291069+0000","last_became_active":"2024-03-30T13:56:51.145567+0000","last_became_peered":"2024-03-30T13:56:51.145567+0000","last_unstale":"2024-03-30T13:57:24.291069+0000","last_undegraded":"2024-03-30T13:57:24.291069+0000","last_fullsized":"2024-03-30T13:57:24.291069+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:25:52.451140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304230+0000","last_change":"2024-03-30T13:56:49.139008+0000","last_active":"2024-03-30T13:57:24.304230+0000","last_peered":"2024-03-30T13:57:24.304230+0000","last_clean":"2024-03-30T13:57:24.304230+0000","last_became_active":"2024-03-30T13:56:49.138730+0000","last_became_peered":"2024-03-30T13:56:49.138730+0000","last_unstale":"2024-03-30T13:57:24.304230+0000","last_undegraded":"2024-03-30T13:57:24.304230+0000","last_fullsized":"2024-03-30T13:57:24.304230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:40:49.910046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291584+0000","last_change":"2024-03-30T13:56:45.114495+0000","last_active":"2024-03-30T13:57:24.291584+0000","last_peered":"2024-03-30T13:57:24.291584+0000","last_clean":"2024-03-30T13:57:24.291584+0000","last_became_active":"2024-03-30T13:56:45.114222+0000","last_became_peered":"2024-03-30T13:56:45.114222+0000","last_unstale":"2024-03-30T13:57:24.291584+0000","last_undegraded":"2024-03-30T13:57:24.291584+0000","last_fullsized":"2024-03-30T13:57:24.291584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:08:44.809377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204262+0000","last_change":"2024-03-30T13:56:47.131000+0000","last_active":"2024-03-30T13:57:24.204262+0000","last_peered":"2024-03-30T13:57:24.204262+0000","last_clean":"2024-03-30T13:57:24.204262+0000","last_became_active":"2024-03-30T13:56:47.130615+0000","last_became_peered":"2024-03-30T13:56:47.130615+0000","last_unstale":"2024-03-30T13:57:24.204262+0000","last_undegraded":"2024-03-30T13:57:24.204262+0000","last_fullsized":"2024-03-30T13:57:24.204262+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:56:00.303907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203645+0000","last_change":"2024-03-30T13:56:51.376529+0000","last_active":"2024-03-30T13:57:24.203645+0000","last_peered":"2024-03-30T13:57:24.203645+0000","last_clean":"2024-03-30T13:57:24.203645+0000","last_became_active":"2024-03-30T13:56:51.376208+0000","last_became_peered":"2024-03-30T13:56:51.376208+0000","last_unstale":"2024-03-30T13:57:24.203645+0000","last_undegraded":"2024-03-30T13:57:24.203645+0000","last_fullsized":"2024-03-30T13:57:24.203645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:33:05.667190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205323+0000","last_change":"2024-03-30T13:56:49.129945+0000","last_active":"2024-03-30T13:57:24.205323+0000","last_peered":"2024-03-30T13:57:24.205323+0000","last_clean":"2024-03-30T13:57:24.205323+0000","last_became_active":"2024-03-30T13:56:49.129553+0000","last_became_peered":"2024-03-30T13:56:49.129553+0000","last_unstale":"2024-03-30T13:57:24.205323+0000","last_undegraded":"2024-03-30T13:57:24.205323+0000","last_fullsized":"2024-03-30T13:57:24.205323+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:29:34.629748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203201+0000","last_change":"2024-03-30T13:56:45.119796+0000","last_active":"2024-03-30T13:57:24.203201+0000","last_peered":"2024-03-30T13:57:24.203201+0000","last_clean":"2024-03-30T13:57:24.203201+0000","last_became_active":"2024-03-30T13:56:45.118039+0000","last_became_peered":"2024-03-30T13:56:45.118039+0000","last_unstale":"2024-03-30T13:57:24.203201+0000","last_undegraded":"2024-03-30T13:57:24.203201+0000","last_fullsized":"2024-03-30T13:57:24.203201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:50:18.766202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1273,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300132+0000","last_change":"2024-03-30T13:56:47.132930+0000","last_active":"2024-03-30T13:57:24.300132+0000","last_peered":"2024-03-30T13:57:24.300132+0000","last_clean":"2024-03-30T13:57:24.300132+0000","last_became_active":"2024-03-30T13:56:47.132351+0000","last_became_peered":"2024-03-30T13:56:47.132351+0000","last_unstale":"2024-03-30T13:57:24.300132+0000","last_undegraded":"2024-03-30T13:57:24.300132+0000","last_fullsized":"2024-03-30T13:57:24.300132+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:21:39.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613410+0000","last_change":"2024-03-30T13:56:51.377257+0000","last_active":"2024-03-30T13:57:24.613410+0000","last_peered":"2024-03-30T13:57:24.613410+0000","last_clean":"2024-03-30T13:57:24.613410+0000","last_became_active":"2024-03-30T13:56:51.376642+0000","last_became_peered":"2024-03-30T13:56:51.376642+0000","last_unstale":"2024-03-30T13:57:24.613410+0000","last_undegraded":"2024-03-30T13:57:24.613410+0000","last_fullsized":"2024-03-30T13:57:24.613410+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:00:02.553486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203538+0000","last_change":"2024-03-30T13:56:49.136898+0000","last_active":"2024-03-30T13:57:24.203538+0000","last_peered":"2024-03-30T13:57:24.203538+0000","last_clean":"2024-03-30T13:57:24.203538+0000","last_became_active":"2024-03-30T13:56:49.136015+0000","last_became_peered":"2024-03-30T13:56:49.136015+0000","last_unstale":"2024-03-30T13:57:24.203538+0000","last_undegraded":"2024-03-30T13:57:24.203538+0000","last_fullsized":"2024-03-30T13:57:24.203538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:56:44.746530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203429+0000","last_change":"2024-03-30T13:56:45.118088+0000","last_active":"2024-03-30T13:57:24.203429+0000","last_peered":"2024-03-30T13:57:24.203429+0000","last_clean":"2024-03-30T13:57:24.203429+0000","last_became_active":"2024-03-30T13:56:45.117779+0000","last_became_peered":"2024-03-30T13:56:45.117779+0000","last_unstale":"2024-03-30T13:57:24.203429+0000","last_undegraded":"2024-03-30T13:57:24.203429+0000","last_fullsized":"2024-03-30T13:57:24.203429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:48:47.526477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613101+0000","last_change":"2024-03-30T13:56:47.134634+0000","last_active":"2024-03-30T13:57:24.613101+0000","last_peered":"2024-03-30T13:57:24.613101+0000","last_clean":"2024-03-30T13:57:24.613101+0000","last_became_active":"2024-03-30T13:56:47.134298+0000","last_became_peered":"2024-03-30T13:56:47.134298+0000","last_unstale":"2024-03-30T13:57:24.613101+0000","last_undegraded":"2024-03-30T13:57:24.613101+0000","last_fullsized":"2024-03-30T13:57:24.613101+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:58:32.338802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303916+0000","last_change":"2024-03-30T13:56:51.144096+0000","last_active":"2024-03-30T13:57:24.303916+0000","last_peered":"2024-03-30T13:57:24.303916+0000","last_clean":"2024-03-30T13:57:24.303916+0000","last_became_active":"2024-03-30T13:56:51.143725+0000","last_became_peered":"2024-03-30T13:56:51.143725+0000","last_unstale":"2024-03-30T13:57:24.303916+0000","last_undegraded":"2024-03-30T13:57:24.303916+0000","last_fullsized":"2024-03-30T13:57:24.303916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:07:38.770131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.175108+0000","last_change":"2024-03-30T13:56:49.139781+0000","last_active":"2024-03-30T13:58:00.175108+0000","last_peered":"2024-03-30T13:58:00.175108+0000","last_clean":"2024-03-30T13:58:00.175108+0000","last_became_active":"2024-03-30T13:56:49.139469+0000","last_became_peered":"2024-03-30T13:56:49.139469+0000","last_unstale":"2024-03-30T13:58:00.175108+0000","last_undegraded":"2024-03-30T13:58:00.175108+0000","last_fullsized":"2024-03-30T13:58:00.175108+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:47:38.713785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204951+0000","last_change":"2024-03-30T13:56:45.123076+0000","last_active":"2024-03-30T13:57:24.204951+0000","last_peered":"2024-03-30T13:57:24.204951+0000","last_clean":"2024-03-30T13:57:24.204951+0000","last_became_active":"2024-03-30T13:56:45.122822+0000","last_became_peered":"2024-03-30T13:56:45.122822+0000","last_unstale":"2024-03-30T13:57:24.204951+0000","last_undegraded":"2024-03-30T13:57:24.204951+0000","last_fullsized":"2024-03-30T13:57:24.204951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:01:36.231878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202642+0000","last_change":"2024-03-30T13:56:48.039417+0000","last_active":"2024-03-30T13:57:24.202642+0000","last_peered":"2024-03-30T13:57:24.202642+0000","last_clean":"2024-03-30T13:57:24.202642+0000","last_became_active":"2024-03-30T13:56:48.039049+0000","last_became_peered":"2024-03-30T13:56:48.039049+0000","last_unstale":"2024-03-30T13:57:24.202642+0000","last_undegraded":"2024-03-30T13:57:24.202642+0000","last_fullsized":"2024-03-30T13:57:24.202642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:19:54.903732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205474+0000","last_change":"2024-03-30T13:56:51.144774+0000","last_active":"2024-03-30T13:57:24.205474+0000","last_peered":"2024-03-30T13:57:24.205474+0000","last_clean":"2024-03-30T13:57:24.205474+0000","last_became_active":"2024-03-30T13:56:51.144421+0000","last_became_peered":"2024-03-30T13:56:51.144421+0000","last_unstale":"2024-03-30T13:57:24.205474+0000","last_undegraded":"2024-03-30T13:57:24.205474+0000","last_fullsized":"2024-03-30T13:57:24.205474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:52:26.415383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202832+0000","last_change":"2024-03-30T13:56:49.134049+0000","last_active":"2024-03-30T13:57:24.202832+0000","last_peered":"2024-03-30T13:57:24.202832+0000","last_clean":"2024-03-30T13:57:24.202832+0000","last_became_active":"2024-03-30T13:56:49.133694+0000","last_became_peered":"2024-03-30T13:56:49.133694+0000","last_unstale":"2024-03-30T13:57:24.202832+0000","last_undegraded":"2024-03-30T13:57:24.202832+0000","last_fullsized":"2024-03-30T13:57:24.202832+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:22:24.597538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612209+0000","last_change":"2024-03-30T13:56:45.118642+0000","last_active":"2024-03-30T13:57:24.612209+0000","last_peered":"2024-03-30T13:57:24.612209+0000","last_clean":"2024-03-30T13:57:24.612209+0000","last_became_active":"2024-03-30T13:56:45.118040+0000","last_became_peered":"2024-03-30T13:56:45.118040+0000","last_unstale":"2024-03-30T13:57:24.612209+0000","last_undegraded":"2024-03-30T13:57:24.612209+0000","last_fullsized":"2024-03-30T13:57:24.612209+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:48:57.536328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.171100+0000","last_change":"2024-03-30T13:56:47.125571+0000","last_active":"2024-03-30T13:57:25.171100+0000","last_peered":"2024-03-30T13:57:25.171100+0000","last_clean":"2024-03-30T13:57:25.171100+0000","last_became_active":"2024-03-30T13:56:47.125192+0000","last_became_peered":"2024-03-30T13:56:47.125192+0000","last_unstale":"2024-03-30T13:57:25.171100+0000","last_undegraded":"2024-03-30T13:57:25.171100+0000","last_fullsized":"2024-03-30T13:57:25.171100+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:44:31.930956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303839+0000","last_change":"2024-03-30T13:56:51.143460+0000","last_active":"2024-03-30T13:57:24.303839+0000","last_peered":"2024-03-30T13:57:24.303839+0000","last_clean":"2024-03-30T13:57:24.303839+0000","last_became_active":"2024-03-30T13:56:51.143137+0000","last_became_peered":"2024-03-30T13:56:51.143137+0000","last_unstale":"2024-03-30T13:57:24.303839+0000","last_undegraded":"2024-03-30T13:57:24.303839+0000","last_fullsized":"2024-03-30T13:57:24.303839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:45:54.882185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.170386+0000","last_change":"2024-03-30T13:56:49.139846+0000","last_active":"2024-03-30T13:57:25.170386+0000","last_peered":"2024-03-30T13:57:25.170386+0000","last_clean":"2024-03-30T13:57:25.170386+0000","last_became_active":"2024-03-30T13:56:49.139589+0000","last_became_peered":"2024-03-30T13:56:49.139589+0000","last_unstale":"2024-03-30T13:57:25.170386+0000","last_undegraded":"2024-03-30T13:57:25.170386+0000","last_fullsized":"2024-03-30T13:57:25.170386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:37:22.628416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202543+0000","last_change":"2024-03-30T13:56:45.120873+0000","last_active":"2024-03-30T13:57:24.202543+0000","last_peered":"2024-03-30T13:57:24.202543+0000","last_clean":"2024-03-30T13:57:24.202543+0000","last_became_active":"2024-03-30T13:56:45.120226+0000","last_became_peered":"2024-03-30T13:56:45.120226+0000","last_unstale":"2024-03-30T13:57:24.202543+0000","last_undegraded":"2024-03-30T13:57:24.202543+0000","last_fullsized":"2024-03-30T13:57:24.202543+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:09:46.349553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612266+0000","last_change":"2024-03-30T13:56:47.128260+0000","last_active":"2024-03-30T13:57:24.612266+0000","last_peered":"2024-03-30T13:57:24.612266+0000","last_clean":"2024-03-30T13:57:24.612266+0000","last_became_active":"2024-03-30T13:56:47.127544+0000","last_became_peered":"2024-03-30T13:56:47.127544+0000","last_unstale":"2024-03-30T13:57:24.612266+0000","last_undegraded":"2024-03-30T13:57:24.612266+0000","last_fullsized":"2024-03-30T13:57:24.612266+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:14:37.168959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612503+0000","last_change":"2024-03-30T13:56:51.376808+0000","last_active":"2024-03-30T13:57:24.612503+0000","last_peered":"2024-03-30T13:57:24.612503+0000","last_clean":"2024-03-30T13:57:24.612503+0000","last_became_active":"2024-03-30T13:56:51.376489+0000","last_became_peered":"2024-03-30T13:56:51.376489+0000","last_unstale":"2024-03-30T13:57:24.612503+0000","last_undegraded":"2024-03-30T13:57:24.612503+0000","last_fullsized":"2024-03-30T13:57:24.612503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:32:42.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304104+0000","last_change":"2024-03-30T13:56:49.139874+0000","last_active":"2024-03-30T13:57:24.304104+0000","last_peered":"2024-03-30T13:57:24.304104+0000","last_clean":"2024-03-30T13:57:24.304104+0000","last_became_active":"2024-03-30T13:56:49.139320+0000","last_became_peered":"2024-03-30T13:56:49.139320+0000","last_unstale":"2024-03-30T13:57:24.304104+0000","last_undegraded":"2024-03-30T13:57:24.304104+0000","last_fullsized":"2024-03-30T13:57:24.304104+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:19:36.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291822+0000","last_change":"2024-03-30T13:56:45.117342+0000","last_active":"2024-03-30T13:57:24.291822+0000","last_peered":"2024-03-30T13:57:24.291822+0000","last_clean":"2024-03-30T13:57:24.291822+0000","last_became_active":"2024-03-30T13:56:45.116838+0000","last_became_peered":"2024-03-30T13:56:45.116838+0000","last_unstale":"2024-03-30T13:57:24.291822+0000","last_undegraded":"2024-03-30T13:57:24.291822+0000","last_fullsized":"2024-03-30T13:57:24.291822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:52:54.379597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612513+0000","last_change":"2024-03-30T13:56:48.037141+0000","last_active":"2024-03-30T13:57:24.612513+0000","last_peered":"2024-03-30T13:57:24.612513+0000","last_clean":"2024-03-30T13:57:24.612513+0000","last_became_active":"2024-03-30T13:56:48.036561+0000","last_became_peered":"2024-03-30T13:56:48.036561+0000","last_unstale":"2024-03-30T13:57:24.612513+0000","last_undegraded":"2024-03-30T13:57:24.612513+0000","last_fullsized":"2024-03-30T13:57:24.612513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:26:50.494560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.201918+0000","last_change":"2024-03-30T13:56:51.146984+0000","last_active":"2024-03-30T13:57:24.201918+0000","last_peered":"2024-03-30T13:57:24.201918+0000","last_clean":"2024-03-30T13:57:24.201918+0000","last_became_active":"2024-03-30T13:56:51.144901+0000","last_became_peered":"2024-03-30T13:56:51.144901+0000","last_unstale":"2024-03-30T13:57:24.201918+0000","last_undegraded":"2024-03-30T13:57:24.201918+0000","last_fullsized":"2024-03-30T13:57:24.201918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:17:05.923333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612799+0000","last_change":"2024-03-30T13:56:49.134564+0000","last_active":"2024-03-30T13:57:24.612799+0000","last_peered":"2024-03-30T13:57:24.612799+0000","last_clean":"2024-03-30T13:57:24.612799+0000","last_became_active":"2024-03-30T13:56:49.133687+0000","last_became_peered":"2024-03-30T13:56:49.133687+0000","last_unstale":"2024-03-30T13:57:24.612799+0000","last_undegraded":"2024-03-30T13:57:24.612799+0000","last_fullsized":"2024-03-30T13:57:24.612799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:17:20.063679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304328+0000","last_change":"2024-03-30T13:56:45.117684+0000","last_active":"2024-03-30T13:57:24.304328+0000","last_peered":"2024-03-30T13:57:24.304328+0000","last_clean":"2024-03-30T13:57:24.304328+0000","last_became_active":"2024-03-30T13:56:45.117377+0000","last_became_peered":"2024-03-30T13:56:45.117377+0000","last_unstale":"2024-03-30T13:57:24.304328+0000","last_undegraded":"2024-03-30T13:57:24.304328+0000","last_fullsized":"2024-03-30T13:57:24.304328+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:28:13.342179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204144+0000","last_change":"2024-03-30T13:56:47.129790+0000","last_active":"2024-03-30T13:57:24.204144+0000","last_peered":"2024-03-30T13:57:24.204144+0000","last_clean":"2024-03-30T13:57:24.204144+0000","last_became_active":"2024-03-30T13:56:47.129477+0000","last_became_peered":"2024-03-30T13:56:47.129477+0000","last_unstale":"2024-03-30T13:57:24.204144+0000","last_undegraded":"2024-03-30T13:57:24.204144+0000","last_fullsized":"2024-03-30T13:57:24.204144+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:00.772680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612284+0000","last_change":"2024-03-30T13:56:51.150812+0000","last_active":"2024-03-30T13:57:24.612284+0000","last_peered":"2024-03-30T13:57:24.612284+0000","last_clean":"2024-03-30T13:57:24.612284+0000","last_became_active":"2024-03-30T13:56:51.150460+0000","last_became_peered":"2024-03-30T13:56:51.150460+0000","last_unstale":"2024-03-30T13:57:24.612284+0000","last_undegraded":"2024-03-30T13:57:24.612284+0000","last_fullsized":"2024-03-30T13:57:24.612284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:48:49.857265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203922+0000","last_change":"2024-03-30T13:56:49.139114+0000","last_active":"2024-03-30T13:57:24.203922+0000","last_peered":"2024-03-30T13:57:24.203922+0000","last_clean":"2024-03-30T13:57:24.203922+0000","last_became_active":"2024-03-30T13:56:49.138750+0000","last_became_peered":"2024-03-30T13:56:49.138750+0000","last_unstale":"2024-03-30T13:57:24.203922+0000","last_undegraded":"2024-03-30T13:57:24.203922+0000","last_fullsized":"2024-03-30T13:57:24.203922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:18:22.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203678+0000","last_change":"2024-03-30T13:56:45.119524+0000","last_active":"2024-03-30T13:57:24.203678+0000","last_peered":"2024-03-30T13:57:24.203678+0000","last_clean":"2024-03-30T13:57:24.203678+0000","last_became_active":"2024-03-30T13:56:45.119230+0000","last_became_peered":"2024-03-30T13:56:45.119230+0000","last_unstale":"2024-03-30T13:57:24.203678+0000","last_undegraded":"2024-03-30T13:57:24.203678+0000","last_fullsized":"2024-03-30T13:57:24.203678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:43:22.784479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204472+0000","last_change":"2024-03-30T13:56:47.120553+0000","last_active":"2024-03-30T13:57:24.204472+0000","last_peered":"2024-03-30T13:57:24.204472+0000","last_clean":"2024-03-30T13:57:24.204472+0000","last_became_active":"2024-03-30T13:56:47.120189+0000","last_became_peered":"2024-03-30T13:56:47.120189+0000","last_unstale":"2024-03-30T13:57:24.204472+0000","last_undegraded":"2024-03-30T13:57:24.204472+0000","last_fullsized":"2024-03-30T13:57:24.204472+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:21:47.977524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202323+0000","last_change":"2024-03-30T13:56:51.151218+0000","last_active":"2024-03-30T13:57:24.202323+0000","last_peered":"2024-03-30T13:57:24.202323+0000","last_clean":"2024-03-30T13:57:24.202323+0000","last_became_active":"2024-03-30T13:56:51.150865+0000","last_became_peered":"2024-03-30T13:56:51.150865+0000","last_unstale":"2024-03-30T13:57:24.202323+0000","last_undegraded":"2024-03-30T13:57:24.202323+0000","last_fullsized":"2024-03-30T13:57:24.202323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:08:27.775166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.175055+0000","last_change":"2024-03-30T13:56:49.140625+0000","last_active":"2024-03-30T13:58:00.175055+0000","last_peered":"2024-03-30T13:58:00.175055+0000","last_clean":"2024-03-30T13:58:00.175055+0000","last_became_active":"2024-03-30T13:56:49.140340+0000","last_became_peered":"2024-03-30T13:56:49.140340+0000","last_unstale":"2024-03-30T13:58:00.175055+0000","last_undegraded":"2024-03-30T13:58:00.175055+0000","last_fullsized":"2024-03-30T13:58:00.175055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:27:01.477794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303961+0000","last_change":"2024-03-30T13:56:45.120035+0000","last_active":"2024-03-30T13:57:24.303961+0000","last_peered":"2024-03-30T13:57:24.303961+0000","last_clean":"2024-03-30T13:57:24.303961+0000","last_became_active":"2024-03-30T13:56:45.119170+0000","last_became_peered":"2024-03-30T13:56:45.119170+0000","last_unstale":"2024-03-30T13:57:24.303961+0000","last_undegraded":"2024-03-30T13:57:24.303961+0000","last_fullsized":"2024-03-30T13:57:24.303961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:50:13.267995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":62,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612770+0000","last_change":"2024-03-30T13:57:09.184977+0000","last_active":"2024-03-30T13:57:24.612770+0000","last_peered":"2024-03-30T13:57:24.612770+0000","last_clean":"2024-03-30T13:57:24.612770+0000","last_became_active":"2024-03-30T13:57:09.173724+0000","last_became_peered":"2024-03-30T13:57:09.173724+0000","last_unstale":"2024-03-30T13:57:24.612770+0000","last_undegraded":"2024-03-30T13:57:24.612770+0000","last_fullsized":"2024-03-30T13:57:24.612770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:41:19.452412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204041+0000","last_change":"2024-03-30T13:56:51.146553+0000","last_active":"2024-03-30T13:57:24.204041+0000","last_peered":"2024-03-30T13:57:24.204041+0000","last_clean":"2024-03-30T13:57:24.204041+0000","last_became_active":"2024-03-30T13:56:51.146205+0000","last_became_peered":"2024-03-30T13:56:51.146205+0000","last_unstale":"2024-03-30T13:57:24.204041+0000","last_undegraded":"2024-03-30T13:57:24.204041+0000","last_fullsized":"2024-03-30T13:57:24.204041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:01:54.831679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":53,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.175240+0000","last_change":"2024-03-30T13:56:49.137965+0000","last_active":"2024-03-30T13:58:00.175240+0000","last_peered":"2024-03-30T13:58:00.175240+0000","last_clean":"2024-03-30T13:58:00.175240+0000","last_became_active":"2024-03-30T13:56:49.137672+0000","last_became_peered":"2024-03-30T13:56:49.137672+0000","last_unstale":"2024-03-30T13:58:00.175240+0000","last_undegraded":"2024-03-30T13:58:00.175240+0000","last_fullsized":"2024-03-30T13:58:00.175240+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:31:40.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202185+0000","last_change":"2024-03-30T13:56:45.120466+0000","last_active":"2024-03-30T13:57:24.202185+0000","last_peered":"2024-03-30T13:57:24.202185+0000","last_clean":"2024-03-30T13:57:24.202185+0000","last_became_active":"2024-03-30T13:56:45.118485+0000","last_became_peered":"2024-03-30T13:56:45.118485+0000","last_unstale":"2024-03-30T13:57:24.202185+0000","last_undegraded":"2024-03-30T13:57:24.202185+0000","last_fullsized":"2024-03-30T13:57:24.202185+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:20:48.928837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291687+0000","last_change":"2024-03-30T13:56:48.039279+0000","last_active":"2024-03-30T13:57:24.291687+0000","last_peered":"2024-03-30T13:57:24.291687+0000","last_clean":"2024-03-30T13:57:24.291687+0000","last_became_active":"2024-03-30T13:56:48.038872+0000","last_became_peered":"2024-03-30T13:56:48.038872+0000","last_unstale":"2024-03-30T13:57:24.291687+0000","last_undegraded":"2024-03-30T13:57:24.291687+0000","last_fullsized":"2024-03-30T13:57:24.291687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:26:27.204441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300365+0000","last_change":"2024-03-30T13:56:51.143070+0000","last_active":"2024-03-30T13:57:24.300365+0000","last_peered":"2024-03-30T13:57:24.300365+0000","last_clean":"2024-03-30T13:57:24.300365+0000","last_became_active":"2024-03-30T13:56:51.142763+0000","last_became_peered":"2024-03-30T13:56:51.142763+0000","last_unstale":"2024-03-30T13:57:24.300365+0000","last_undegraded":"2024-03-30T13:57:24.300365+0000","last_fullsized":"2024-03-30T13:57:24.300365+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:02:53.647876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613387+0000","last_change":"2024-03-30T13:56:49.139217+0000","last_active":"2024-03-30T13:57:24.613387+0000","last_peered":"2024-03-30T13:57:24.613387+0000","last_clean":"2024-03-30T13:57:24.613387+0000","last_became_active":"2024-03-30T13:56:49.138891+0000","last_became_peered":"2024-03-30T13:56:49.138891+0000","last_unstale":"2024-03-30T13:57:24.613387+0000","last_undegraded":"2024-03-30T13:57:24.613387+0000","last_fullsized":"2024-03-30T13:57:24.613387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:46:50.528300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304043+0000","last_change":"2024-03-30T13:56:45.116615+0000","last_active":"2024-03-30T13:57:24.304043+0000","last_peered":"2024-03-30T13:57:24.304043+0000","last_clean":"2024-03-30T13:57:24.304043+0000","last_became_active":"2024-03-30T13:56:45.116334+0000","last_became_peered":"2024-03-30T13:56:45.116334+0000","last_unstale":"2024-03-30T13:57:24.304043+0000","last_undegraded":"2024-03-30T13:57:24.304043+0000","last_fullsized":"2024-03-30T13:57:24.304043+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:30:13.716951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202717+0000","last_change":"2024-03-30T13:56:48.036822+0000","last_active":"2024-03-30T13:57:24.202717+0000","last_peered":"2024-03-30T13:57:24.202717+0000","last_clean":"2024-03-30T13:57:24.202717+0000","last_became_active":"2024-03-30T13:56:48.036471+0000","last_became_peered":"2024-03-30T13:56:48.036471+0000","last_unstale":"2024-03-30T13:57:24.202717+0000","last_undegraded":"2024-03-30T13:57:24.202717+0000","last_fullsized":"2024-03-30T13:57:24.202717+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:58:01.993816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304054+0000","last_change":"2024-03-30T13:56:51.376420+0000","last_active":"2024-03-30T13:57:24.304054+0000","last_peered":"2024-03-30T13:57:24.304054+0000","last_clean":"2024-03-30T13:57:24.304054+0000","last_became_active":"2024-03-30T13:56:51.375902+0000","last_became_peered":"2024-03-30T13:56:51.375902+0000","last_unstale":"2024-03-30T13:57:24.304054+0000","last_undegraded":"2024-03-30T13:57:24.304054+0000","last_fullsized":"2024-03-30T13:57:24.304054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:16:19.572835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291632+0000","last_change":"2024-03-30T13:56:49.143704+0000","last_active":"2024-03-30T13:57:24.291632+0000","last_peered":"2024-03-30T13:57:24.291632+0000","last_clean":"2024-03-30T13:57:24.291632+0000","last_became_active":"2024-03-30T13:56:49.143393+0000","last_became_peered":"2024-03-30T13:56:49.143393+0000","last_unstale":"2024-03-30T13:57:24.291632+0000","last_undegraded":"2024-03-30T13:57:24.291632+0000","last_fullsized":"2024-03-30T13:57:24.291632+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:51:23.903350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.169934+0000","last_change":"2024-03-30T13:56:45.122014+0000","last_active":"2024-03-30T13:57:25.169934+0000","last_peered":"2024-03-30T13:57:25.169934+0000","last_clean":"2024-03-30T13:57:25.169934+0000","last_became_active":"2024-03-30T13:56:45.121587+0000","last_became_peered":"2024-03-30T13:56:45.121587+0000","last_unstale":"2024-03-30T13:57:25.169934+0000","last_undegraded":"2024-03-30T13:57:25.169934+0000","last_fullsized":"2024-03-30T13:57:25.169934+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:02:50.921093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613476+0000","last_change":"2024-03-30T13:56:48.037393+0000","last_active":"2024-03-30T13:57:24.613476+0000","last_peered":"2024-03-30T13:57:24.613476+0000","last_clean":"2024-03-30T13:57:24.613476+0000","last_became_active":"2024-03-30T13:56:48.037084+0000","last_became_peered":"2024-03-30T13:56:48.037084+0000","last_unstale":"2024-03-30T13:57:24.613476+0000","last_undegraded":"2024-03-30T13:57:24.613476+0000","last_fullsized":"2024-03-30T13:57:24.613476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:38:43.687720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291704+0000","last_change":"2024-03-30T13:56:51.374229+0000","last_active":"2024-03-30T13:57:24.291704+0000","last_peered":"2024-03-30T13:57:24.291704+0000","last_clean":"2024-03-30T13:57:24.291704+0000","last_became_active":"2024-03-30T13:56:51.373906+0000","last_became_peered":"2024-03-30T13:56:51.373906+0000","last_unstale":"2024-03-30T13:57:24.291704+0000","last_undegraded":"2024-03-30T13:57:24.291704+0000","last_fullsized":"2024-03-30T13:57:24.291704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:45:25.296390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304335+0000","last_change":"2024-03-30T13:56:45.122862+0000","last_active":"2024-03-30T13:57:24.304335+0000","last_peered":"2024-03-30T13:57:24.304335+0000","last_clean":"2024-03-30T13:57:24.304335+0000","last_became_active":"2024-03-30T13:56:45.122609+0000","last_became_peered":"2024-03-30T13:56:45.122609+0000","last_unstale":"2024-03-30T13:57:24.304335+0000","last_undegraded":"2024-03-30T13:57:24.304335+0000","last_fullsized":"2024-03-30T13:57:24.304335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:33:11.217111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291156+0000","last_change":"2024-03-30T13:56:48.039492+0000","last_active":"2024-03-30T13:57:24.291156+0000","last_peered":"2024-03-30T13:57:24.291156+0000","last_clean":"2024-03-30T13:57:24.291156+0000","last_became_active":"2024-03-30T13:56:48.038898+0000","last_became_peered":"2024-03-30T13:56:48.038898+0000","last_unstale":"2024-03-30T13:57:24.291156+0000","last_undegraded":"2024-03-30T13:57:24.291156+0000","last_fullsized":"2024-03-30T13:57:24.291156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:52:14.536138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.175065+0000","last_change":"2024-03-30T13:56:49.140187+0000","last_active":"2024-03-30T13:58:00.175065+0000","last_peered":"2024-03-30T13:58:00.175065+0000","last_clean":"2024-03-30T13:58:00.175065+0000","last_became_active":"2024-03-30T13:56:49.139903+0000","last_became_peered":"2024-03-30T13:56:49.139903+0000","last_unstale":"2024-03-30T13:58:00.175065+0000","last_undegraded":"2024-03-30T13:58:00.175065+0000","last_fullsized":"2024-03-30T13:58:00.175065+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:34:22.725256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612610+0000","last_change":"2024-03-30T13:56:51.146611+0000","last_active":"2024-03-30T13:57:24.612610+0000","last_peered":"2024-03-30T13:57:24.612610+0000","last_clean":"2024-03-30T13:57:24.612610+0000","last_became_active":"2024-03-30T13:56:51.146277+0000","last_became_peered":"2024-03-30T13:56:51.146277+0000","last_unstale":"2024-03-30T13:57:24.612610+0000","last_undegraded":"2024-03-30T13:57:24.612610+0000","last_fullsized":"2024-03-30T13:57:24.612610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:08:12.492187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304360+0000","last_change":"2024-03-30T13:56:45.118302+0000","last_active":"2024-03-30T13:57:24.304360+0000","last_peered":"2024-03-30T13:57:24.304360+0000","last_clean":"2024-03-30T13:57:24.304360+0000","last_became_active":"2024-03-30T13:56:45.117976+0000","last_became_peered":"2024-03-30T13:56:45.117976+0000","last_unstale":"2024-03-30T13:57:24.304360+0000","last_undegraded":"2024-03-30T13:57:24.304360+0000","last_fullsized":"2024-03-30T13:57:24.304360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:07:02.148184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203408+0000","last_change":"2024-03-30T13:57:09.179801+0000","last_active":"2024-03-30T13:57:24.203408+0000","last_peered":"2024-03-30T13:57:24.203408+0000","last_clean":"2024-03-30T13:57:24.203408+0000","last_became_active":"2024-03-30T13:57:09.173514+0000","last_became_peered":"2024-03-30T13:57:09.173514+0000","last_unstale":"2024-03-30T13:57:24.203408+0000","last_undegraded":"2024-03-30T13:57:24.203408+0000","last_fullsized":"2024-03-30T13:57:24.203408+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:54:56.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.299618+0000","last_change":"2024-03-30T13:56:49.134000+0000","last_active":"2024-03-30T13:57:24.299618+0000","last_peered":"2024-03-30T13:57:24.299618+0000","last_clean":"2024-03-30T13:57:24.299618+0000","last_became_active":"2024-03-30T13:56:49.133135+0000","last_became_peered":"2024-03-30T13:56:49.133135+0000","last_unstale":"2024-03-30T13:57:24.299618+0000","last_undegraded":"2024-03-30T13:57:24.299618+0000","last_fullsized":"2024-03-30T13:57:24.299618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:04:33.457010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303652+0000","last_change":"2024-03-30T13:56:51.147026+0000","last_active":"2024-03-30T13:57:24.303652+0000","last_peered":"2024-03-30T13:57:24.303652+0000","last_clean":"2024-03-30T13:57:24.303652+0000","last_became_active":"2024-03-30T13:56:51.146720+0000","last_became_peered":"2024-03-30T13:56:51.146720+0000","last_unstale":"2024-03-30T13:57:24.303652+0000","last_undegraded":"2024-03-30T13:57:24.303652+0000","last_fullsized":"2024-03-30T13:57:24.303652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:55:05.576387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.299958+0000","last_change":"2024-03-30T13:56:49.137678+0000","last_active":"2024-03-30T13:57:24.299958+0000","last_peered":"2024-03-30T13:57:24.299958+0000","last_clean":"2024-03-30T13:57:24.299958+0000","last_became_active":"2024-03-30T13:56:49.137366+0000","last_became_peered":"2024-03-30T13:56:49.137366+0000","last_unstale":"2024-03-30T13:57:24.299958+0000","last_undegraded":"2024-03-30T13:57:24.299958+0000","last_fullsized":"2024-03-30T13:57:24.299958+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:05:57.360946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304500+0000","last_change":"2024-03-30T13:56:45.120209+0000","last_active":"2024-03-30T13:57:24.304500+0000","last_peered":"2024-03-30T13:57:24.304500+0000","last_clean":"2024-03-30T13:57:24.304500+0000","last_became_active":"2024-03-30T13:56:45.119107+0000","last_became_peered":"2024-03-30T13:56:45.119107+0000","last_unstale":"2024-03-30T13:57:24.304500+0000","last_undegraded":"2024-03-30T13:57:24.304500+0000","last_fullsized":"2024-03-30T13:57:24.304500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:35:12.748914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.170147+0000","last_change":"2024-03-30T13:57:09.151659+0000","last_active":"2024-03-30T13:57:25.170147+0000","last_peered":"2024-03-30T13:57:25.170147+0000","last_clean":"2024-03-30T13:57:25.170147+0000","last_became_active":"2024-03-30T13:57:09.141903+0000","last_became_peered":"2024-03-30T13:57:09.141903+0000","last_unstale":"2024-03-30T13:57:25.170147+0000","last_undegraded":"2024-03-30T13:57:25.170147+0000","last_fullsized":"2024-03-30T13:57:25.170147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:10:57.530537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.201980+0000","last_change":"2024-03-30T13:56:51.147843+0000","last_active":"2024-03-30T13:57:24.201980+0000","last_peered":"2024-03-30T13:57:24.201980+0000","last_clean":"2024-03-30T13:57:24.201980+0000","last_became_active":"2024-03-30T13:56:51.147505+0000","last_became_peered":"2024-03-30T13:56:51.147505+0000","last_unstale":"2024-03-30T13:57:24.201980+0000","last_undegraded":"2024-03-30T13:57:24.201980+0000","last_fullsized":"2024-03-30T13:57:24.201980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:22:39.554642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300002+0000","last_change":"2024-03-30T13:56:49.140253+0000","last_active":"2024-03-30T13:57:24.300002+0000","last_peered":"2024-03-30T13:57:24.300002+0000","last_clean":"2024-03-30T13:57:24.300002+0000","last_became_active":"2024-03-30T13:56:49.139900+0000","last_became_peered":"2024-03-30T13:56:49.139900+0000","last_unstale":"2024-03-30T13:57:24.300002+0000","last_undegraded":"2024-03-30T13:57:24.300002+0000","last_fullsized":"2024-03-30T13:57:24.300002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:05:17.865193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291618+0000","last_change":"2024-03-30T13:56:45.119791+0000","last_active":"2024-03-30T13:57:24.291618+0000","last_peered":"2024-03-30T13:57:24.291618+0000","last_clean":"2024-03-30T13:57:24.291618+0000","last_became_active":"2024-03-30T13:56:45.119497+0000","last_became_peered":"2024-03-30T13:56:45.119497+0000","last_unstale":"2024-03-30T13:57:24.291618+0000","last_undegraded":"2024-03-30T13:57:24.291618+0000","last_fullsized":"2024-03-30T13:57:24.291618+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:27:07.256388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203503+0000","last_change":"2024-03-30T13:56:47.120630+0000","last_active":"2024-03-30T13:57:24.203503+0000","last_peered":"2024-03-30T13:57:24.203503+0000","last_clean":"2024-03-30T13:57:24.203503+0000","last_became_active":"2024-03-30T13:56:47.120245+0000","last_became_peered":"2024-03-30T13:56:47.120245+0000","last_unstale":"2024-03-30T13:57:24.203503+0000","last_undegraded":"2024-03-30T13:57:24.203503+0000","last_fullsized":"2024-03-30T13:57:24.203503+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:13:52.849073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613157+0000","last_change":"2024-03-30T13:56:51.144198+0000","last_active":"2024-03-30T13:57:24.613157+0000","last_peered":"2024-03-30T13:57:24.613157+0000","last_clean":"2024-03-30T13:57:24.613157+0000","last_became_active":"2024-03-30T13:56:51.143783+0000","last_became_peered":"2024-03-30T13:56:51.143783+0000","last_unstale":"2024-03-30T13:57:24.613157+0000","last_undegraded":"2024-03-30T13:57:24.613157+0000","last_fullsized":"2024-03-30T13:57:24.613157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:09:38.179728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291210+0000","last_change":"2024-03-30T13:56:48.120221+0000","last_active":"2024-03-30T13:57:24.291210+0000","last_peered":"2024-03-30T13:57:24.291210+0000","last_clean":"2024-03-30T13:57:24.291210+0000","last_became_active":"2024-03-30T13:56:44.113778+0000","last_became_peered":"2024-03-30T13:56:44.113778+0000","last_unstale":"2024-03-30T13:57:24.291210+0000","last_undegraded":"2024-03-30T13:57:24.291210+0000","last_fullsized":"2024-03-30T13:57:24.291210+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_clean_scrub_stamp":"2024-03-30T13:56:43.098310+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:48:43.293005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051844900000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203303+0000","last_change":"2024-03-30T13:56:45.120368+0000","last_active":"2024-03-30T13:57:24.203303+0000","last_peered":"2024-03-30T13:57:24.203303+0000","last_clean":"2024-03-30T13:57:24.203303+0000","last_became_active":"2024-03-30T13:56:45.120077+0000","last_became_peered":"2024-03-30T13:56:45.120077+0000","last_unstale":"2024-03-30T13:57:24.203303+0000","last_undegraded":"2024-03-30T13:57:24.203303+0000","last_fullsized":"2024-03-30T13:57:24.203303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:22:03.966315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":94,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.174933+0000","last_change":"2024-03-30T13:56:47.131477+0000","last_active":"2024-03-30T13:58:00.174933+0000","last_peered":"2024-03-30T13:58:00.174933+0000","last_clean":"2024-03-30T13:58:00.174933+0000","last_became_active":"2024-03-30T13:56:47.131044+0000","last_became_peered":"2024-03-30T13:56:47.131044+0000","last_unstale":"2024-03-30T13:58:00.174933+0000","last_undegraded":"2024-03-30T13:58:00.174933+0000","last_fullsized":"2024-03-30T13:58:00.174933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:48:16.154473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202317+0000","last_change":"2024-03-30T13:56:49.130328+0000","last_active":"2024-03-30T13:57:24.202317+0000","last_peered":"2024-03-30T13:57:24.202317+0000","last_clean":"2024-03-30T13:57:24.202317+0000","last_became_active":"2024-03-30T13:56:49.130031+0000","last_became_peered":"2024-03-30T13:56:49.130031+0000","last_unstale":"2024-03-30T13:57:24.202317+0000","last_undegraded":"2024-03-30T13:57:24.202317+0000","last_fullsized":"2024-03-30T13:57:24.202317+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:36:51.024682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304554+0000","last_change":"2024-03-30T13:56:51.374278+0000","last_active":"2024-03-30T13:57:24.304554+0000","last_peered":"2024-03-30T13:57:24.304554+0000","last_clean":"2024-03-30T13:57:24.304554+0000","last_became_active":"2024-03-30T13:56:51.373977+0000","last_became_peered":"2024-03-30T13:56:51.373977+0000","last_unstale":"2024-03-30T13:57:24.304554+0000","last_undegraded":"2024-03-30T13:57:24.304554+0000","last_fullsized":"2024-03-30T13:57:24.304554+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:24:58.288537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203105+0000","last_change":"2024-03-30T13:56:45.116511+0000","last_active":"2024-03-30T13:57:24.203105+0000","last_peered":"2024-03-30T13:57:24.203105+0000","last_clean":"2024-03-30T13:57:24.203105+0000","last_became_active":"2024-03-30T13:56:45.116189+0000","last_became_peered":"2024-03-30T13:56:45.116189+0000","last_unstale":"2024-03-30T13:57:24.203105+0000","last_undegraded":"2024-03-30T13:57:24.203105+0000","last_fullsized":"2024-03-30T13:57:24.203105+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:06:00.495646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.299765+0000","last_change":"2024-03-30T13:56:48.119693+0000","last_active":"2024-03-30T13:57:24.299765+0000","last_peered":"2024-03-30T13:57:24.299765+0000","last_clean":"2024-03-30T13:57:24.299765+0000","last_became_active":"2024-03-30T13:56:44.115568+0000","last_became_peered":"2024-03-30T13:56:44.115568+0000","last_unstale":"2024-03-30T13:57:24.299765+0000","last_undegraded":"2024-03-30T13:57:24.299765+0000","last_fullsized":"2024-03-30T13:57:24.299765+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_clean_scrub_stamp":"2024-03-30T13:56:43.098310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:01:25.483974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001808111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205747+0000","last_change":"2024-03-30T13:56:47.124567+0000","last_active":"2024-03-30T13:57:24.205747+0000","last_peered":"2024-03-30T13:57:24.205747+0000","last_clean":"2024-03-30T13:57:24.205747+0000","last_became_active":"2024-03-30T13:56:47.124221+0000","last_became_peered":"2024-03-30T13:56:47.124221+0000","last_unstale":"2024-03-30T13:57:24.205747+0000","last_undegraded":"2024-03-30T13:57:24.205747+0000","last_fullsized":"2024-03-30T13:57:24.205747+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:28:29.960992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300359+0000","last_change":"2024-03-30T13:56:49.138374+0000","last_active":"2024-03-30T13:57:24.300359+0000","last_peered":"2024-03-30T13:57:24.300359+0000","last_clean":"2024-03-30T13:57:24.300359+0000","last_became_active":"2024-03-30T13:56:49.137770+0000","last_became_peered":"2024-03-30T13:56:49.137770+0000","last_unstale":"2024-03-30T13:57:24.300359+0000","last_undegraded":"2024-03-30T13:57:24.300359+0000","last_fullsized":"2024-03-30T13:57:24.300359+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:48:05.776781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291484+0000","last_change":"2024-03-30T13:56:51.145093+0000","last_active":"2024-03-30T13:57:24.291484+0000","last_peered":"2024-03-30T13:57:24.291484+0000","last_clean":"2024-03-30T13:57:24.291484+0000","last_became_active":"2024-03-30T13:56:51.144742+0000","last_became_peered":"2024-03-30T13:56:51.144742+0000","last_unstale":"2024-03-30T13:57:24.291484+0000","last_undegraded":"2024-03-30T13:57:24.291484+0000","last_fullsized":"2024-03-30T13:57:24.291484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:01:03.127914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:26.294665+0000","last_change":"2024-03-30T13:56:31.734197+0000","last_active":"2024-03-30T13:57:26.294665+0000","last_peered":"2024-03-30T13:57:26.294665+0000","last_clean":"2024-03-30T13:57:26.294665+0000","last_became_active":"2024-03-30T13:56:31.733805+0000","last_became_peered":"2024-03-30T13:56:31.733805+0000","last_unstale":"2024-03-30T13:57:26.294665+0000","last_undegraded":"2024-03-30T13:57:26.294665+0000","last_fullsized":"2024-03-30T13:57:26.294665+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:54:25.686692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:54:25.686692+0000","last_clean_scrub_stamp":"2024-03-30T13:54:25.686692+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:13:07.005289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612740+0000","last_change":"2024-03-30T13:56:45.121763+0000","last_active":"2024-03-30T13:57:24.612740+0000","last_peered":"2024-03-30T13:57:24.612740+0000","last_clean":"2024-03-30T13:57:24.612740+0000","last_became_active":"2024-03-30T13:56:45.121424+0000","last_became_peered":"2024-03-30T13:56:45.121424+0000","last_unstale":"2024-03-30T13:57:24.612740+0000","last_undegraded":"2024-03-30T13:57:24.612740+0000","last_fullsized":"2024-03-30T13:57:24.612740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:29:50.114112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":68,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.175184+0000","last_change":"2024-03-30T13:56:48.038014+0000","last_active":"2024-03-30T13:58:00.175184+0000","last_peered":"2024-03-30T13:58:00.175184+0000","last_clean":"2024-03-30T13:58:00.175184+0000","last_became_active":"2024-03-30T13:56:48.037609+0000","last_became_peered":"2024-03-30T13:56:48.037609+0000","last_unstale":"2024-03-30T13:58:00.175184+0000","last_undegraded":"2024-03-30T13:58:00.175184+0000","last_fullsized":"2024-03-30T13:58:00.175184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:46:30.568119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304199+0000","last_change":"2024-03-30T13:56:49.137841+0000","last_active":"2024-03-30T13:57:24.304199+0000","last_peered":"2024-03-30T13:57:24.304199+0000","last_clean":"2024-03-30T13:57:24.304199+0000","last_became_active":"2024-03-30T13:56:49.137576+0000","last_became_peered":"2024-03-30T13:56:49.137576+0000","last_unstale":"2024-03-30T13:57:24.304199+0000","last_undegraded":"2024-03-30T13:57:24.304199+0000","last_fullsized":"2024-03-30T13:57:24.304199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:03:10.485823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204549+0000","last_change":"2024-03-30T13:56:51.148056+0000","last_active":"2024-03-30T13:57:24.204549+0000","last_peered":"2024-03-30T13:57:24.204549+0000","last_clean":"2024-03-30T13:57:24.204549+0000","last_became_active":"2024-03-30T13:56:51.147616+0000","last_became_peered":"2024-03-30T13:56:51.147616+0000","last_unstale":"2024-03-30T13:57:24.204549+0000","last_undegraded":"2024-03-30T13:57:24.204549+0000","last_fullsized":"2024-03-30T13:57:24.204549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:44:43.140295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202156+0000","last_change":"2024-03-30T13:56:45.116207+0000","last_active":"2024-03-30T13:57:24.202156+0000","last_peered":"2024-03-30T13:57:24.202156+0000","last_clean":"2024-03-30T13:57:24.202156+0000","last_became_active":"2024-03-30T13:56:45.115723+0000","last_became_peered":"2024-03-30T13:56:45.115723+0000","last_unstale":"2024-03-30T13:57:24.202156+0000","last_undegraded":"2024-03-30T13:57:24.202156+0000","last_fullsized":"2024-03-30T13:57:24.202156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:28:05.722637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":118,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:04.157470+0000","last_change":"2024-03-30T13:56:48.120793+0000","last_active":"2024-03-30T13:58:04.157470+0000","last_peered":"2024-03-30T13:58:04.157470+0000","last_clean":"2024-03-30T13:58:04.157470+0000","last_became_active":"2024-03-30T13:56:44.117022+0000","last_became_peered":"2024-03-30T13:56:44.117022+0000","last_unstale":"2024-03-30T13:58:04.157470+0000","last_undegraded":"2024-03-30T13:58:04.157470+0000","last_fullsized":"2024-03-30T13:58:04.157470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_clean_scrub_stamp":"2024-03-30T13:56:43.098310+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:37:47.279259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093502400000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202481+0000","last_change":"2024-03-30T13:56:47.124453+0000","last_active":"2024-03-30T13:57:24.202481+0000","last_peered":"2024-03-30T13:57:24.202481+0000","last_clean":"2024-03-30T13:57:24.202481+0000","last_became_active":"2024-03-30T13:56:47.124164+0000","last_became_peered":"2024-03-30T13:56:47.124164+0000","last_unstale":"2024-03-30T13:57:24.202481+0000","last_undegraded":"2024-03-30T13:57:24.202481+0000","last_fullsized":"2024-03-30T13:57:24.202481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:27:29.282192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291756+0000","last_change":"2024-03-30T13:56:49.141989+0000","last_active":"2024-03-30T13:57:24.291756+0000","last_peered":"2024-03-30T13:57:24.291756+0000","last_clean":"2024-03-30T13:57:24.291756+0000","last_became_active":"2024-03-30T13:56:49.141714+0000","last_became_peered":"2024-03-30T13:56:49.141714+0000","last_unstale":"2024-03-30T13:57:24.291756+0000","last_undegraded":"2024-03-30T13:57:24.291756+0000","last_fullsized":"2024-03-30T13:57:24.291756+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:58:01.263979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202767+0000","last_change":"2024-03-30T13:56:51.376844+0000","last_active":"2024-03-30T13:57:24.202767+0000","last_peered":"2024-03-30T13:57:24.202767+0000","last_clean":"2024-03-30T13:57:24.202767+0000","last_became_active":"2024-03-30T13:56:51.376490+0000","last_became_peered":"2024-03-30T13:56:51.376490+0000","last_unstale":"2024-03-30T13:57:24.202767+0000","last_undegraded":"2024-03-30T13:57:24.202767+0000","last_fullsized":"2024-03-30T13:57:24.202767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:05:39.060308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204754+0000","last_change":"2024-03-30T13:56:45.116114+0000","last_active":"2024-03-30T13:57:24.204754+0000","last_peered":"2024-03-30T13:57:24.204754+0000","last_clean":"2024-03-30T13:57:24.204754+0000","last_became_active":"2024-03-30T13:56:45.115817+0000","last_became_peered":"2024-03-30T13:56:45.115817+0000","last_unstale":"2024-03-30T13:57:24.204754+0000","last_undegraded":"2024-03-30T13:57:24.204754+0000","last_fullsized":"2024-03-30T13:57:24.204754+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:02:48.206255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203308+0000","last_change":"2024-03-30T13:56:48.036836+0000","last_active":"2024-03-30T13:57:24.203308+0000","last_peered":"2024-03-30T13:57:24.203308+0000","last_clean":"2024-03-30T13:57:24.203308+0000","last_became_active":"2024-03-30T13:56:48.036484+0000","last_became_peered":"2024-03-30T13:56:48.036484+0000","last_unstale":"2024-03-30T13:57:24.203308+0000","last_undegraded":"2024-03-30T13:57:24.203308+0000","last_fullsized":"2024-03-30T13:57:24.203308+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:06:31.777370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.170753+0000","last_change":"2024-03-30T13:56:49.138264+0000","last_active":"2024-03-30T13:57:25.170753+0000","last_peered":"2024-03-30T13:57:25.170753+0000","last_clean":"2024-03-30T13:57:25.170753+0000","last_became_active":"2024-03-30T13:56:49.137954+0000","last_became_peered":"2024-03-30T13:56:49.137954+0000","last_unstale":"2024-03-30T13:57:25.170753+0000","last_undegraded":"2024-03-30T13:57:25.170753+0000","last_fullsized":"2024-03-30T13:57:25.170753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:31:01.533147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.201918+0000","last_change":"2024-03-30T13:56:51.149024+0000","last_active":"2024-03-30T13:57:24.201918+0000","last_peered":"2024-03-30T13:57:24.201918+0000","last_clean":"2024-03-30T13:57:24.201918+0000","last_became_active":"2024-03-30T13:56:51.148658+0000","last_became_peered":"2024-03-30T13:56:51.148658+0000","last_unstale":"2024-03-30T13:57:24.201918+0000","last_undegraded":"2024-03-30T13:57:24.201918+0000","last_fullsized":"2024-03-30T13:57:24.201918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:56:49.675264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291525+0000","last_change":"2024-03-30T13:56:45.116248+0000","last_active":"2024-03-30T13:57:24.291525+0000","last_peered":"2024-03-30T13:57:24.291525+0000","last_clean":"2024-03-30T13:57:24.291525+0000","last_became_active":"2024-03-30T13:56:45.115966+0000","last_became_peered":"2024-03-30T13:56:45.115966+0000","last_unstale":"2024-03-30T13:57:24.291525+0000","last_undegraded":"2024-03-30T13:57:24.291525+0000","last_fullsized":"2024-03-30T13:57:24.291525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:35:48.959141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204025+0000","last_change":"2024-03-30T13:56:47.125784+0000","last_active":"2024-03-30T13:57:24.204025+0000","last_peered":"2024-03-30T13:57:24.204025+0000","last_clean":"2024-03-30T13:57:24.204025+0000","last_became_active":"2024-03-30T13:56:47.125437+0000","last_became_peered":"2024-03-30T13:56:47.125437+0000","last_unstale":"2024-03-30T13:57:24.204025+0000","last_undegraded":"2024-03-30T13:57:24.204025+0000","last_fullsized":"2024-03-30T13:57:24.204025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:16:32.723025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.174938+0000","last_change":"2024-03-30T13:56:49.136739+0000","last_active":"2024-03-30T13:58:00.174938+0000","last_peered":"2024-03-30T13:58:00.174938+0000","last_clean":"2024-03-30T13:58:00.174938+0000","last_became_active":"2024-03-30T13:56:49.136450+0000","last_became_peered":"2024-03-30T13:56:49.136450+0000","last_unstale":"2024-03-30T13:58:00.174938+0000","last_undegraded":"2024-03-30T13:58:00.174938+0000","last_fullsized":"2024-03-30T13:58:00.174938+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:35:40.590636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303831+0000","last_change":"2024-03-30T13:56:51.376269+0000","last_active":"2024-03-30T13:57:24.303831+0000","last_peered":"2024-03-30T13:57:24.303831+0000","last_clean":"2024-03-30T13:57:24.303831+0000","last_became_active":"2024-03-30T13:56:51.375831+0000","last_became_peered":"2024-03-30T13:56:51.375831+0000","last_unstale":"2024-03-30T13:57:24.303831+0000","last_undegraded":"2024-03-30T13:57:24.303831+0000","last_fullsized":"2024-03-30T13:57:24.303831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:44.828337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202614+0000","last_change":"2024-03-30T13:56:45.115397+0000","last_active":"2024-03-30T13:57:24.202614+0000","last_peered":"2024-03-30T13:57:24.202614+0000","last_clean":"2024-03-30T13:57:24.202614+0000","last_became_active":"2024-03-30T13:56:45.115143+0000","last_became_peered":"2024-03-30T13:56:45.115143+0000","last_unstale":"2024-03-30T13:57:24.202614+0000","last_undegraded":"2024-03-30T13:57:24.202614+0000","last_fullsized":"2024-03-30T13:57:24.202614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:08:22.202665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203385+0000","last_change":"2024-03-30T13:56:47.118382+0000","last_active":"2024-03-30T13:57:24.203385+0000","last_peered":"2024-03-30T13:57:24.203385+0000","last_clean":"2024-03-30T13:57:24.203385+0000","last_became_active":"2024-03-30T13:56:47.117854+0000","last_became_peered":"2024-03-30T13:56:47.117854+0000","last_unstale":"2024-03-30T13:57:24.203385+0000","last_undegraded":"2024-03-30T13:57:24.203385+0000","last_fullsized":"2024-03-30T13:57:24.203385+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:33:09.207059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.174933+0000","last_change":"2024-03-30T13:56:49.138280+0000","last_active":"2024-03-30T13:58:00.174933+0000","last_peered":"2024-03-30T13:58:00.174933+0000","last_clean":"2024-03-30T13:58:00.174933+0000","last_became_active":"2024-03-30T13:56:49.137991+0000","last_became_peered":"2024-03-30T13:56:49.137991+0000","last_unstale":"2024-03-30T13:58:00.174933+0000","last_undegraded":"2024-03-30T13:58:00.174933+0000","last_fullsized":"2024-03-30T13:58:00.174933+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:53:27.179411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203226+0000","last_change":"2024-03-30T13:56:51.146969+0000","last_active":"2024-03-30T13:57:24.203226+0000","last_peered":"2024-03-30T13:57:24.203226+0000","last_clean":"2024-03-30T13:57:24.203226+0000","last_became_active":"2024-03-30T13:56:51.146674+0000","last_became_peered":"2024-03-30T13:56:51.146674+0000","last_unstale":"2024-03-30T13:57:24.203226+0000","last_undegraded":"2024-03-30T13:57:24.203226+0000","last_fullsized":"2024-03-30T13:57:24.203226+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:12:14.805690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613279+0000","last_change":"2024-03-30T13:56:45.119767+0000","last_active":"2024-03-30T13:57:24.613279+0000","last_peered":"2024-03-30T13:57:24.613279+0000","last_clean":"2024-03-30T13:57:24.613279+0000","last_became_active":"2024-03-30T13:56:45.119379+0000","last_became_peered":"2024-03-30T13:56:45.119379+0000","last_unstale":"2024-03-30T13:57:24.613279+0000","last_undegraded":"2024-03-30T13:57:24.613279+0000","last_fullsized":"2024-03-30T13:57:24.613279+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:41.829117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612513+0000","last_change":"2024-03-30T13:56:48.039657+0000","last_active":"2024-03-30T13:57:24.612513+0000","last_peered":"2024-03-30T13:57:24.612513+0000","last_clean":"2024-03-30T13:57:24.612513+0000","last_became_active":"2024-03-30T13:56:48.039120+0000","last_became_peered":"2024-03-30T13:56:48.039120+0000","last_unstale":"2024-03-30T13:57:24.612513+0000","last_undegraded":"2024-03-30T13:57:24.612513+0000","last_fullsized":"2024-03-30T13:57:24.612513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:16:16.597222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203150+0000","last_change":"2024-03-30T13:56:49.135107+0000","last_active":"2024-03-30T13:57:24.203150+0000","last_peered":"2024-03-30T13:57:24.203150+0000","last_clean":"2024-03-30T13:57:24.203150+0000","last_became_active":"2024-03-30T13:56:49.133957+0000","last_became_peered":"2024-03-30T13:56:49.133957+0000","last_unstale":"2024-03-30T13:57:24.203150+0000","last_undegraded":"2024-03-30T13:57:24.203150+0000","last_fullsized":"2024-03-30T13:57:24.203150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:43:55.232949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291361+0000","last_change":"2024-03-30T13:56:51.144275+0000","last_active":"2024-03-30T13:57:24.291361+0000","last_peered":"2024-03-30T13:57:24.291361+0000","last_clean":"2024-03-30T13:57:24.291361+0000","last_became_active":"2024-03-30T13:56:51.143965+0000","last_became_peered":"2024-03-30T13:56:51.143965+0000","last_unstale":"2024-03-30T13:57:24.291361+0000","last_undegraded":"2024-03-30T13:57:24.291361+0000","last_fullsized":"2024-03-30T13:57:24.291361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:25:23.436503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.175069+0000","last_change":"2024-03-30T13:56:49.137904+0000","last_active":"2024-03-30T13:58:00.175069+0000","last_peered":"2024-03-30T13:58:00.175069+0000","last_clean":"2024-03-30T13:58:00.175069+0000","last_became_active":"2024-03-30T13:56:49.137627+0000","last_became_peered":"2024-03-30T13:56:49.137627+0000","last_unstale":"2024-03-30T13:58:00.175069+0000","last_undegraded":"2024-03-30T13:58:00.175069+0000","last_fullsized":"2024-03-30T13:58:00.175069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:53:23.470116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612402+0000","last_change":"2024-03-30T13:56:45.118344+0000","last_active":"2024-03-30T13:57:24.612402+0000","last_peered":"2024-03-30T13:57:24.612402+0000","last_clean":"2024-03-30T13:57:24.612402+0000","last_became_active":"2024-03-30T13:56:45.117633+0000","last_became_peered":"2024-03-30T13:56:45.117633+0000","last_unstale":"2024-03-30T13:57:24.612402+0000","last_undegraded":"2024-03-30T13:57:24.612402+0000","last_fullsized":"2024-03-30T13:57:24.612402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:57:19.902679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205806+0000","last_change":"2024-03-30T13:56:47.127609+0000","last_active":"2024-03-30T13:57:24.205806+0000","last_peered":"2024-03-30T13:57:24.205806+0000","last_clean":"2024-03-30T13:57:24.205806+0000","last_became_active":"2024-03-30T13:56:47.126134+0000","last_became_peered":"2024-03-30T13:56:47.126134+0000","last_unstale":"2024-03-30T13:57:24.205806+0000","last_undegraded":"2024-03-30T13:57:24.205806+0000","last_fullsized":"2024-03-30T13:57:24.205806+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:53:09.400906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612581+0000","last_change":"2024-03-30T13:56:51.374114+0000","last_active":"2024-03-30T13:57:24.612581+0000","last_peered":"2024-03-30T13:57:24.612581+0000","last_clean":"2024-03-30T13:57:24.612581+0000","last_became_active":"2024-03-30T13:56:51.373775+0000","last_became_peered":"2024-03-30T13:56:51.373775+0000","last_unstale":"2024-03-30T13:57:24.612581+0000","last_undegraded":"2024-03-30T13:57:24.612581+0000","last_fullsized":"2024-03-30T13:57:24.612581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:32:18.819817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291692+0000","last_change":"2024-03-30T13:56:49.142752+0000","last_active":"2024-03-30T13:57:24.291692+0000","last_peered":"2024-03-30T13:57:24.291692+0000","last_clean":"2024-03-30T13:57:24.291692+0000","last_became_active":"2024-03-30T13:56:49.142488+0000","last_became_peered":"2024-03-30T13:56:49.142488+0000","last_unstale":"2024-03-30T13:57:24.291692+0000","last_undegraded":"2024-03-30T13:57:24.291692+0000","last_fullsized":"2024-03-30T13:57:24.291692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:07:26.493606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203256+0000","last_change":"2024-03-30T13:56:45.116294+0000","last_active":"2024-03-30T13:57:24.203256+0000","last_peered":"2024-03-30T13:57:24.203256+0000","last_clean":"2024-03-30T13:57:24.203256+0000","last_became_active":"2024-03-30T13:56:45.115982+0000","last_became_peered":"2024-03-30T13:56:45.115982+0000","last_unstale":"2024-03-30T13:57:24.203256+0000","last_undegraded":"2024-03-30T13:57:24.203256+0000","last_fullsized":"2024-03-30T13:57:24.203256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:26:49.056274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203866+0000","last_change":"2024-03-30T13:56:47.121770+0000","last_active":"2024-03-30T13:57:24.203866+0000","last_peered":"2024-03-30T13:57:24.203866+0000","last_clean":"2024-03-30T13:57:24.203866+0000","last_became_active":"2024-03-30T13:56:47.121315+0000","last_became_peered":"2024-03-30T13:56:47.121315+0000","last_unstale":"2024-03-30T13:57:24.203866+0000","last_undegraded":"2024-03-30T13:57:24.203866+0000","last_fullsized":"2024-03-30T13:57:24.203866+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:42:17.898358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300677+0000","last_change":"2024-03-30T13:56:51.144168+0000","last_active":"2024-03-30T13:57:24.300677+0000","last_peered":"2024-03-30T13:57:24.300677+0000","last_clean":"2024-03-30T13:57:24.300677+0000","last_became_active":"2024-03-30T13:56:51.143711+0000","last_became_peered":"2024-03-30T13:56:51.143711+0000","last_unstale":"2024-03-30T13:57:24.300677+0000","last_undegraded":"2024-03-30T13:57:24.300677+0000","last_fullsized":"2024-03-30T13:57:24.300677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:57:23.216165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204990+0000","last_change":"2024-03-30T13:56:49.129026+0000","last_active":"2024-03-30T13:57:24.204990+0000","last_peered":"2024-03-30T13:57:24.204990+0000","last_clean":"2024-03-30T13:57:24.204990+0000","last_became_active":"2024-03-30T13:56:49.128710+0000","last_became_peered":"2024-03-30T13:56:49.128710+0000","last_unstale":"2024-03-30T13:57:24.204990+0000","last_undegraded":"2024-03-30T13:57:24.204990+0000","last_fullsized":"2024-03-30T13:57:24.204990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:09:59.892110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.169965+0000","last_change":"2024-03-30T13:56:45.121108+0000","last_active":"2024-03-30T13:57:25.169965+0000","last_peered":"2024-03-30T13:57:25.169965+0000","last_clean":"2024-03-30T13:57:25.169965+0000","last_became_active":"2024-03-30T13:56:45.120823+0000","last_became_peered":"2024-03-30T13:56:45.120823+0000","last_unstale":"2024-03-30T13:57:25.169965+0000","last_undegraded":"2024-03-30T13:57:25.169965+0000","last_fullsized":"2024-03-30T13:57:25.169965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:36:10.942984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204371+0000","last_change":"2024-03-30T13:56:47.126611+0000","last_active":"2024-03-30T13:57:24.204371+0000","last_peered":"2024-03-30T13:57:24.204371+0000","last_clean":"2024-03-30T13:57:24.204371+0000","last_became_active":"2024-03-30T13:56:47.126287+0000","last_became_peered":"2024-03-30T13:56:47.126287+0000","last_unstale":"2024-03-30T13:57:24.204371+0000","last_undegraded":"2024-03-30T13:57:24.204371+0000","last_fullsized":"2024-03-30T13:57:24.204371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:54:23.267905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204619+0000","last_change":"2024-03-30T13:56:51.147528+0000","last_active":"2024-03-30T13:57:24.204619+0000","last_peered":"2024-03-30T13:57:24.204619+0000","last_clean":"2024-03-30T13:57:24.204619+0000","last_became_active":"2024-03-30T13:56:51.147147+0000","last_became_peered":"2024-03-30T13:56:51.147147+0000","last_unstale":"2024-03-30T13:57:24.204619+0000","last_undegraded":"2024-03-30T13:57:24.204619+0000","last_fullsized":"2024-03-30T13:57:24.204619+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:23:07.720769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:00.174934+0000","last_change":"2024-03-30T13:56:49.136870+0000","last_active":"2024-03-30T13:58:00.174934+0000","last_peered":"2024-03-30T13:58:00.174934+0000","last_clean":"2024-03-30T13:58:00.174934+0000","last_became_active":"2024-03-30T13:56:49.136538+0000","last_became_peered":"2024-03-30T13:56:49.136538+0000","last_unstale":"2024-03-30T13:58:00.174934+0000","last_undegraded":"2024-03-30T13:58:00.174934+0000","last_fullsized":"2024-03-30T13:58:00.174934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:58:15.275356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612886+0000","last_change":"2024-03-30T13:56:45.119857+0000","last_active":"2024-03-30T13:57:24.612886+0000","last_peered":"2024-03-30T13:57:24.612886+0000","last_clean":"2024-03-30T13:57:24.612886+0000","last_became_active":"2024-03-30T13:56:45.119543+0000","last_became_peered":"2024-03-30T13:56:45.119543+0000","last_unstale":"2024-03-30T13:57:24.612886+0000","last_undegraded":"2024-03-30T13:57:24.612886+0000","last_fullsized":"2024-03-30T13:57:24.612886+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:14:58.774300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203580+0000","last_change":"2024-03-30T13:56:47.130841+0000","last_active":"2024-03-30T13:57:24.203580+0000","last_peered":"2024-03-30T13:57:24.203580+0000","last_clean":"2024-03-30T13:57:24.203580+0000","last_became_active":"2024-03-30T13:56:47.130484+0000","last_became_peered":"2024-03-30T13:56:47.130484+0000","last_unstale":"2024-03-30T13:57:24.203580+0000","last_undegraded":"2024-03-30T13:57:24.203580+0000","last_fullsized":"2024-03-30T13:57:24.203580+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:59:40.892832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1318912,"data_stored":873616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":871291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":749568,"data_stored":282378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":873358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":704512,"data_stored":281271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":737280,"data_stored":281827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":737280,"data_stored":282814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":704512,"data_stored":281359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:58:09.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T13:58:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:10.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:10.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:10.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.293 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T13:58:10.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:58:10.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:10 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:11.040 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:10 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:10.898Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000190122s 2024-03-30T13:58:11.322 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:11 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:58:11.322 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:11 smithi102 ceph-mon[29392]: Deploying daemon grafana.a on smithi138 2024-03-30T13:58:11.323 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:11 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:58:11.323 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:11 smithi102 ceph-mon[37717]: Deploying daemon grafana.a on smithi138 2024-03-30T13:58:11.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:11 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T13:58:11.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:11 smithi138 ceph-mon[33397]: Deploying daemon grafana.a on smithi138 2024-03-30T13:58:11.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 -- 172.21.15.102:0/1981806441 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 msgr2=0x7f335410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:11.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 --2- 172.21.15.102:0/1981806441 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 0x7f335410dfc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f334c009990 tx=0x7f334c009ca0 comp rx=0 tx=0).stop 2024-03-30T13:58:11.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 -- 172.21.15.102:0/1981806441 shutdown_connections 2024-03-30T13:58:11.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 --2- 172.21.15.102:0/1981806441 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 0x7f335410dfc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:11.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 --2- 172.21.15.102:0/1981806441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 0x7f3354107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:11.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 --2- 172.21.15.102:0/1981806441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 0x7f33541015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:11.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 -- 172.21.15.102:0/1981806441 >> 172.21.15.102:0/1981806441 conn(0x7f3354078070 msgr2=0x7f3354078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:11.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 -- 172.21.15.102:0/1981806441 shutdown_connections 2024-03-30T13:58:11.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.998+0000 7f3359846700 1 -- 172.21.15.102:0/1981806441 wait complete. 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 Processor -- start 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 -- start start 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 0x7f335410ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 0x7f335410f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 0x7f335410f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3354113e40 con 0x7f3354101140 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3354113cc0 con 0x7f3354107180 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:11.999+0000 7f3359846700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3354113fc0 con 0x7f3354101ae0 2024-03-30T13:58:11.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 0x7f335410ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f33537fe700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 0x7f335410f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f33537fe700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 0x7f335410f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.102:54344/0 (socket says 172.21.15.102:54344) 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f33537fe700 1 -- 172.21.15.102:0/1769406034 learned_addr learned my addr 172.21.15.102:0/1769406034 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 0x7f335410ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46906/0 (socket says 172.21.15.102:46906) 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f33527fc700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 0x7f335410f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 -- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 msgr2=0x7f335410f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 0x7f335410f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 -- 172.21.15.102:0/1769406034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 msgr2=0x7f335410f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 0x7f335410f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 -- 172.21.15.102:0/1769406034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33541aa910 con 0x7f3354101140 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f33537fe700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 0x7f335410f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3352ffd700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 0x7f335410ec80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f334400ef10 tx=0x7f334400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.000+0000 7f3358844700 1 -- 172.21.15.102:0/1769406034 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334400c9c0 con 0x7f3354101140 2024-03-30T13:58:11.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.001+0000 7f3359846700 1 -- 172.21.15.102:0/1769406034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33541aaba0 con 0x7f3354101140 2024-03-30T13:58:11.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.001+0000 7f3358844700 1 -- 172.21.15.102:0/1769406034 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f3344019070 con 0x7f3354101140 2024-03-30T13:58:11.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.001+0000 7f3358844700 1 -- 172.21.15.102:0/1769406034 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344014480 con 0x7f3354101140 2024-03-30T13:58:11.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.001+0000 7f3359846700 1 -- 172.21.15.102:0/1769406034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33541ab130 con 0x7f3354101140 2024-03-30T13:58:12.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.002+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3334005120 con 0x7f3354101140 2024-03-30T13:58:12.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.002+0000 7f3358844700 1 -- 172.21.15.102:0/1769406034 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f3344012050 con 0x7f3354101140 2024-03-30T13:58:12.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.003+0000 7f3358844700 1 --2- 172.21.15.102:0/1769406034 >> v2:172.21.15.102:6800/622906077 conn(0x7f333c06dbf0 0x7f333c0700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:12.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.003+0000 7f33527fc700 1 --2- 172.21.15.102:0/1769406034 >> v2:172.21.15.102:6800/622906077 conn(0x7f333c06dbf0 0x7f333c0700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:12.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.003+0000 7f3358844700 1 -- 172.21.15.102:0/1769406034 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7f334408da70 con 0x7f3354101140 2024-03-30T13:58:12.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.003+0000 7f33527fc700 1 --2- 172.21.15.102:0/1769406034 >> v2:172.21.15.102:6800/622906077 conn(0x7f333c06dbf0 0x7f333c0700b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3348009680 tx=0x7f3348009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:12.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.008+0000 7f3358844700 1 -- 172.21.15.102:0/1769406034 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3344010040 con 0x7f3354101140 2024-03-30T13:58:12.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.199+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 --> v2:172.21.15.102:6800/622906077 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3334000bc0 con 0x7f333c06dbf0 2024-03-30T13:58:12.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.208+0000 7f3358844700 1 -- 172.21.15.102:0/1769406034 <== mgr.24464 v2:172.21.15.102:6800/622906077 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346464 (secure 0 0 0) 0x7f3334000bc0 con 0x7f333c06dbf0 2024-03-30T13:58:12.209 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 >> v2:172.21.15.102:6800/622906077 conn(0x7f333c06dbf0 msgr2=0x7f333c0700b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 --2- 172.21.15.102:0/1769406034 >> v2:172.21.15.102:6800/622906077 conn(0x7f333c06dbf0 0x7f333c0700b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3348009680 tx=0x7f3348009260 comp rx=0 tx=0).stop 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 msgr2=0x7f335410ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 0x7f335410ec80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f334400ef10 tx=0x7f334400c3a0 comp rx=0 tx=0).stop 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 shutdown_connections 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3354101140 0x7f335410ec80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 --2- 172.21.15.102:0/1769406034 >> v2:172.21.15.102:6800/622906077 conn(0x7f333c06dbf0 0x7f333c0700b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3354107180 0x7f335410f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 --2- 172.21.15.102:0/1769406034 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3354101ae0 0x7f335410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 >> 172.21.15.102:0/1769406034 conn(0x7f3354078070 msgr2=0x7f3354104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 shutdown_connections 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:12.213+0000 7f333a7fc700 1 -- 172.21.15.102:0/1769406034 wait complete. 2024-03-30T13:58:12.212 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-30T13:58:12.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:12 smithi138 ceph-mon[33397]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:12.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:12 smithi102 ceph-mon[29392]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:12.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:12 smithi102 ceph-mon[37717]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:12.863 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-30T13:58:10.233611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1012,"num_read_kb":858,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240180,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721036,"statfs":{"total":767960285184,"available":767714340864,"internally_reserved":0,"allocated":7565312,"data_stored":4027914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003115"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203242+0000","last_change":"2024-03-30T13:56:51.145179+0000","last_active":"2024-03-30T13:57:24.203242+0000","last_peered":"2024-03-30T13:57:24.203242+0000","last_clean":"2024-03-30T13:57:24.203242+0000","last_became_active":"2024-03-30T13:56:51.144878+0000","last_became_peered":"2024-03-30T13:56:51.144878+0000","last_unstale":"2024-03-30T13:57:24.203242+0000","last_undegraded":"2024-03-30T13:57:24.203242+0000","last_fullsized":"2024-03-30T13:57:24.203242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:34:38.183139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204856+0000","last_change":"2024-03-30T13:56:49.138594+0000","last_active":"2024-03-30T13:57:24.204856+0000","last_peered":"2024-03-30T13:57:24.204856+0000","last_clean":"2024-03-30T13:57:24.204856+0000","last_became_active":"2024-03-30T13:56:49.138275+0000","last_became_peered":"2024-03-30T13:56:49.138275+0000","last_unstale":"2024-03-30T13:57:24.204856+0000","last_undegraded":"2024-03-30T13:57:24.204856+0000","last_fullsized":"2024-03-30T13:57:24.204856+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:15:00.924078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202536+0000","last_change":"2024-03-30T13:56:45.111202+0000","last_active":"2024-03-30T13:57:24.202536+0000","last_peered":"2024-03-30T13:57:24.202536+0000","last_clean":"2024-03-30T13:57:24.202536+0000","last_became_active":"2024-03-30T13:56:45.110871+0000","last_became_peered":"2024-03-30T13:56:45.110871+0000","last_unstale":"2024-03-30T13:57:24.202536+0000","last_undegraded":"2024-03-30T13:57:24.202536+0000","last_fullsized":"2024-03-30T13:57:24.202536+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:11:00.993985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203097+0000","last_change":"2024-03-30T13:56:47.122055+0000","last_active":"2024-03-30T13:57:24.203097+0000","last_peered":"2024-03-30T13:57:24.203097+0000","last_clean":"2024-03-30T13:57:24.203097+0000","last_became_active":"2024-03-30T13:56:47.121290+0000","last_became_peered":"2024-03-30T13:56:47.121290+0000","last_unstale":"2024-03-30T13:57:24.203097+0000","last_undegraded":"2024-03-30T13:57:24.203097+0000","last_fullsized":"2024-03-30T13:57:24.203097+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:57:40.768021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612707+0000","last_change":"2024-03-30T13:56:51.377451+0000","last_active":"2024-03-30T13:57:24.612707+0000","last_peered":"2024-03-30T13:57:24.612707+0000","last_clean":"2024-03-30T13:57:24.612707+0000","last_became_active":"2024-03-30T13:56:51.376678+0000","last_became_peered":"2024-03-30T13:56:51.376678+0000","last_unstale":"2024-03-30T13:57:24.612707+0000","last_undegraded":"2024-03-30T13:57:24.612707+0000","last_fullsized":"2024-03-30T13:57:24.612707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:46:01.897415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203097+0000","last_change":"2024-03-30T13:56:49.137668+0000","last_active":"2024-03-30T13:57:24.203097+0000","last_peered":"2024-03-30T13:57:24.203097+0000","last_clean":"2024-03-30T13:57:24.203097+0000","last_became_active":"2024-03-30T13:56:49.137322+0000","last_became_peered":"2024-03-30T13:56:49.137322+0000","last_unstale":"2024-03-30T13:57:24.203097+0000","last_undegraded":"2024-03-30T13:57:24.203097+0000","last_fullsized":"2024-03-30T13:57:24.203097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:46:03.272319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613263+0000","last_change":"2024-03-30T13:56:45.121812+0000","last_active":"2024-03-30T13:57:24.613263+0000","last_peered":"2024-03-30T13:57:24.613263+0000","last_clean":"2024-03-30T13:57:24.613263+0000","last_became_active":"2024-03-30T13:56:45.121401+0000","last_became_peered":"2024-03-30T13:56:45.121401+0000","last_unstale":"2024-03-30T13:57:24.613263+0000","last_undegraded":"2024-03-30T13:57:24.613263+0000","last_fullsized":"2024-03-30T13:57:24.613263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:55:32.205894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612606+0000","last_change":"2024-03-30T13:56:47.129602+0000","last_active":"2024-03-30T13:57:24.612606+0000","last_peered":"2024-03-30T13:57:24.612606+0000","last_clean":"2024-03-30T13:57:24.612606+0000","last_became_active":"2024-03-30T13:56:47.129289+0000","last_became_peered":"2024-03-30T13:56:47.129289+0000","last_unstale":"2024-03-30T13:57:24.612606+0000","last_undegraded":"2024-03-30T13:57:24.612606+0000","last_fullsized":"2024-03-30T13:57:24.612606+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:27:19.864234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202206+0000","last_change":"2024-03-30T13:56:51.147121+0000","last_active":"2024-03-30T13:57:24.202206+0000","last_peered":"2024-03-30T13:57:24.202206+0000","last_clean":"2024-03-30T13:57:24.202206+0000","last_became_active":"2024-03-30T13:56:51.145364+0000","last_became_peered":"2024-03-30T13:56:51.145364+0000","last_unstale":"2024-03-30T13:57:24.202206+0000","last_undegraded":"2024-03-30T13:57:24.202206+0000","last_fullsized":"2024-03-30T13:57:24.202206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:41:10.749407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291339+0000","last_change":"2024-03-30T13:56:49.141397+0000","last_active":"2024-03-30T13:57:24.291339+0000","last_peered":"2024-03-30T13:57:24.291339+0000","last_clean":"2024-03-30T13:57:24.291339+0000","last_became_active":"2024-03-30T13:56:49.141111+0000","last_became_peered":"2024-03-30T13:56:49.141111+0000","last_unstale":"2024-03-30T13:57:24.291339+0000","last_undegraded":"2024-03-30T13:57:24.291339+0000","last_fullsized":"2024-03-30T13:57:24.291339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:16:27.094352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291434+0000","last_change":"2024-03-30T13:56:45.121753+0000","last_active":"2024-03-30T13:57:24.291434+0000","last_peered":"2024-03-30T13:57:24.291434+0000","last_clean":"2024-03-30T13:57:24.291434+0000","last_became_active":"2024-03-30T13:56:45.120978+0000","last_became_peered":"2024-03-30T13:56:45.120978+0000","last_unstale":"2024-03-30T13:57:24.291434+0000","last_undegraded":"2024-03-30T13:57:24.291434+0000","last_fullsized":"2024-03-30T13:57:24.291434+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:54:43.154356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203716+0000","last_change":"2024-03-30T13:56:47.133118+0000","last_active":"2024-03-30T13:57:24.203716+0000","last_peered":"2024-03-30T13:57:24.203716+0000","last_clean":"2024-03-30T13:57:24.203716+0000","last_became_active":"2024-03-30T13:56:47.132851+0000","last_became_peered":"2024-03-30T13:56:47.132851+0000","last_unstale":"2024-03-30T13:57:24.203716+0000","last_undegraded":"2024-03-30T13:57:24.203716+0000","last_fullsized":"2024-03-30T13:57:24.203716+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:31:33.961265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291069+0000","last_change":"2024-03-30T13:56:51.145874+0000","last_active":"2024-03-30T13:57:24.291069+0000","last_peered":"2024-03-30T13:57:24.291069+0000","last_clean":"2024-03-30T13:57:24.291069+0000","last_became_active":"2024-03-30T13:56:51.145567+0000","last_became_peered":"2024-03-30T13:56:51.145567+0000","last_unstale":"2024-03-30T13:57:24.291069+0000","last_undegraded":"2024-03-30T13:57:24.291069+0000","last_fullsized":"2024-03-30T13:57:24.291069+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:25:52.451140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304230+0000","last_change":"2024-03-30T13:56:49.139008+0000","last_active":"2024-03-30T13:57:24.304230+0000","last_peered":"2024-03-30T13:57:24.304230+0000","last_clean":"2024-03-30T13:57:24.304230+0000","last_became_active":"2024-03-30T13:56:49.138730+0000","last_became_peered":"2024-03-30T13:56:49.138730+0000","last_unstale":"2024-03-30T13:57:24.304230+0000","last_undegraded":"2024-03-30T13:57:24.304230+0000","last_fullsized":"2024-03-30T13:57:24.304230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:40:49.910046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291584+0000","last_change":"2024-03-30T13:56:45.114495+0000","last_active":"2024-03-30T13:57:24.291584+0000","last_peered":"2024-03-30T13:57:24.291584+0000","last_clean":"2024-03-30T13:57:24.291584+0000","last_became_active":"2024-03-30T13:56:45.114222+0000","last_became_peered":"2024-03-30T13:56:45.114222+0000","last_unstale":"2024-03-30T13:57:24.291584+0000","last_undegraded":"2024-03-30T13:57:24.291584+0000","last_fullsized":"2024-03-30T13:57:24.291584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:08:44.809377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204262+0000","last_change":"2024-03-30T13:56:47.131000+0000","last_active":"2024-03-30T13:57:24.204262+0000","last_peered":"2024-03-30T13:57:24.204262+0000","last_clean":"2024-03-30T13:57:24.204262+0000","last_became_active":"2024-03-30T13:56:47.130615+0000","last_became_peered":"2024-03-30T13:56:47.130615+0000","last_unstale":"2024-03-30T13:57:24.204262+0000","last_undegraded":"2024-03-30T13:57:24.204262+0000","last_fullsized":"2024-03-30T13:57:24.204262+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:56:00.303907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203645+0000","last_change":"2024-03-30T13:56:51.376529+0000","last_active":"2024-03-30T13:57:24.203645+0000","last_peered":"2024-03-30T13:57:24.203645+0000","last_clean":"2024-03-30T13:57:24.203645+0000","last_became_active":"2024-03-30T13:56:51.376208+0000","last_became_peered":"2024-03-30T13:56:51.376208+0000","last_unstale":"2024-03-30T13:57:24.203645+0000","last_undegraded":"2024-03-30T13:57:24.203645+0000","last_fullsized":"2024-03-30T13:57:24.203645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:33:05.667190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205323+0000","last_change":"2024-03-30T13:56:49.129945+0000","last_active":"2024-03-30T13:57:24.205323+0000","last_peered":"2024-03-30T13:57:24.205323+0000","last_clean":"2024-03-30T13:57:24.205323+0000","last_became_active":"2024-03-30T13:56:49.129553+0000","last_became_peered":"2024-03-30T13:56:49.129553+0000","last_unstale":"2024-03-30T13:57:24.205323+0000","last_undegraded":"2024-03-30T13:57:24.205323+0000","last_fullsized":"2024-03-30T13:57:24.205323+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:29:34.629748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203201+0000","last_change":"2024-03-30T13:56:45.119796+0000","last_active":"2024-03-30T13:57:24.203201+0000","last_peered":"2024-03-30T13:57:24.203201+0000","last_clean":"2024-03-30T13:57:24.203201+0000","last_became_active":"2024-03-30T13:56:45.118039+0000","last_became_peered":"2024-03-30T13:56:45.118039+0000","last_unstale":"2024-03-30T13:57:24.203201+0000","last_undegraded":"2024-03-30T13:57:24.203201+0000","last_fullsized":"2024-03-30T13:57:24.203201+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:50:18.766202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1273,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300132+0000","last_change":"2024-03-30T13:56:47.132930+0000","last_active":"2024-03-30T13:57:24.300132+0000","last_peered":"2024-03-30T13:57:24.300132+0000","last_clean":"2024-03-30T13:57:24.300132+0000","last_became_active":"2024-03-30T13:56:47.132351+0000","last_became_peered":"2024-03-30T13:56:47.132351+0000","last_unstale":"2024-03-30T13:57:24.300132+0000","last_undegraded":"2024-03-30T13:57:24.300132+0000","last_fullsized":"2024-03-30T13:57:24.300132+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:21:39.124588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613410+0000","last_change":"2024-03-30T13:56:51.377257+0000","last_active":"2024-03-30T13:57:24.613410+0000","last_peered":"2024-03-30T13:57:24.613410+0000","last_clean":"2024-03-30T13:57:24.613410+0000","last_became_active":"2024-03-30T13:56:51.376642+0000","last_became_peered":"2024-03-30T13:56:51.376642+0000","last_unstale":"2024-03-30T13:57:24.613410+0000","last_undegraded":"2024-03-30T13:57:24.613410+0000","last_fullsized":"2024-03-30T13:57:24.613410+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:00:02.553486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203538+0000","last_change":"2024-03-30T13:56:49.136898+0000","last_active":"2024-03-30T13:57:24.203538+0000","last_peered":"2024-03-30T13:57:24.203538+0000","last_clean":"2024-03-30T13:57:24.203538+0000","last_became_active":"2024-03-30T13:56:49.136015+0000","last_became_peered":"2024-03-30T13:56:49.136015+0000","last_unstale":"2024-03-30T13:57:24.203538+0000","last_undegraded":"2024-03-30T13:57:24.203538+0000","last_fullsized":"2024-03-30T13:57:24.203538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:56:44.746530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203429+0000","last_change":"2024-03-30T13:56:45.118088+0000","last_active":"2024-03-30T13:57:24.203429+0000","last_peered":"2024-03-30T13:57:24.203429+0000","last_clean":"2024-03-30T13:57:24.203429+0000","last_became_active":"2024-03-30T13:56:45.117779+0000","last_became_peered":"2024-03-30T13:56:45.117779+0000","last_unstale":"2024-03-30T13:57:24.203429+0000","last_undegraded":"2024-03-30T13:57:24.203429+0000","last_fullsized":"2024-03-30T13:57:24.203429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:48:47.526477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613101+0000","last_change":"2024-03-30T13:56:47.134634+0000","last_active":"2024-03-30T13:57:24.613101+0000","last_peered":"2024-03-30T13:57:24.613101+0000","last_clean":"2024-03-30T13:57:24.613101+0000","last_became_active":"2024-03-30T13:56:47.134298+0000","last_became_peered":"2024-03-30T13:56:47.134298+0000","last_unstale":"2024-03-30T13:57:24.613101+0000","last_undegraded":"2024-03-30T13:57:24.613101+0000","last_fullsized":"2024-03-30T13:57:24.613101+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:58:32.338802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303916+0000","last_change":"2024-03-30T13:56:51.144096+0000","last_active":"2024-03-30T13:57:24.303916+0000","last_peered":"2024-03-30T13:57:24.303916+0000","last_clean":"2024-03-30T13:57:24.303916+0000","last_became_active":"2024-03-30T13:56:51.143725+0000","last_became_peered":"2024-03-30T13:56:51.143725+0000","last_unstale":"2024-03-30T13:57:24.303916+0000","last_undegraded":"2024-03-30T13:57:24.303916+0000","last_fullsized":"2024-03-30T13:57:24.303916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:07:38.770131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.176058+0000","last_change":"2024-03-30T13:56:49.139781+0000","last_active":"2024-03-30T13:58:05.176058+0000","last_peered":"2024-03-30T13:58:05.176058+0000","last_clean":"2024-03-30T13:58:05.176058+0000","last_became_active":"2024-03-30T13:56:49.139469+0000","last_became_peered":"2024-03-30T13:56:49.139469+0000","last_unstale":"2024-03-30T13:58:05.176058+0000","last_undegraded":"2024-03-30T13:58:05.176058+0000","last_fullsized":"2024-03-30T13:58:05.176058+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:47:38.713785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204951+0000","last_change":"2024-03-30T13:56:45.123076+0000","last_active":"2024-03-30T13:57:24.204951+0000","last_peered":"2024-03-30T13:57:24.204951+0000","last_clean":"2024-03-30T13:57:24.204951+0000","last_became_active":"2024-03-30T13:56:45.122822+0000","last_became_peered":"2024-03-30T13:56:45.122822+0000","last_unstale":"2024-03-30T13:57:24.204951+0000","last_undegraded":"2024-03-30T13:57:24.204951+0000","last_fullsized":"2024-03-30T13:57:24.204951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:01:36.231878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202642+0000","last_change":"2024-03-30T13:56:48.039417+0000","last_active":"2024-03-30T13:57:24.202642+0000","last_peered":"2024-03-30T13:57:24.202642+0000","last_clean":"2024-03-30T13:57:24.202642+0000","last_became_active":"2024-03-30T13:56:48.039049+0000","last_became_peered":"2024-03-30T13:56:48.039049+0000","last_unstale":"2024-03-30T13:57:24.202642+0000","last_undegraded":"2024-03-30T13:57:24.202642+0000","last_fullsized":"2024-03-30T13:57:24.202642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:19:54.903732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205474+0000","last_change":"2024-03-30T13:56:51.144774+0000","last_active":"2024-03-30T13:57:24.205474+0000","last_peered":"2024-03-30T13:57:24.205474+0000","last_clean":"2024-03-30T13:57:24.205474+0000","last_became_active":"2024-03-30T13:56:51.144421+0000","last_became_peered":"2024-03-30T13:56:51.144421+0000","last_unstale":"2024-03-30T13:57:24.205474+0000","last_undegraded":"2024-03-30T13:57:24.205474+0000","last_fullsized":"2024-03-30T13:57:24.205474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:52:26.415383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202832+0000","last_change":"2024-03-30T13:56:49.134049+0000","last_active":"2024-03-30T13:57:24.202832+0000","last_peered":"2024-03-30T13:57:24.202832+0000","last_clean":"2024-03-30T13:57:24.202832+0000","last_became_active":"2024-03-30T13:56:49.133694+0000","last_became_peered":"2024-03-30T13:56:49.133694+0000","last_unstale":"2024-03-30T13:57:24.202832+0000","last_undegraded":"2024-03-30T13:57:24.202832+0000","last_fullsized":"2024-03-30T13:57:24.202832+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:22:24.597538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612209+0000","last_change":"2024-03-30T13:56:45.118642+0000","last_active":"2024-03-30T13:57:24.612209+0000","last_peered":"2024-03-30T13:57:24.612209+0000","last_clean":"2024-03-30T13:57:24.612209+0000","last_became_active":"2024-03-30T13:56:45.118040+0000","last_became_peered":"2024-03-30T13:56:45.118040+0000","last_unstale":"2024-03-30T13:57:24.612209+0000","last_undegraded":"2024-03-30T13:57:24.612209+0000","last_fullsized":"2024-03-30T13:57:24.612209+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:48:57.536328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.171100+0000","last_change":"2024-03-30T13:56:47.125571+0000","last_active":"2024-03-30T13:57:25.171100+0000","last_peered":"2024-03-30T13:57:25.171100+0000","last_clean":"2024-03-30T13:57:25.171100+0000","last_became_active":"2024-03-30T13:56:47.125192+0000","last_became_peered":"2024-03-30T13:56:47.125192+0000","last_unstale":"2024-03-30T13:57:25.171100+0000","last_undegraded":"2024-03-30T13:57:25.171100+0000","last_fullsized":"2024-03-30T13:57:25.171100+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:44:31.930956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303839+0000","last_change":"2024-03-30T13:56:51.143460+0000","last_active":"2024-03-30T13:57:24.303839+0000","last_peered":"2024-03-30T13:57:24.303839+0000","last_clean":"2024-03-30T13:57:24.303839+0000","last_became_active":"2024-03-30T13:56:51.143137+0000","last_became_peered":"2024-03-30T13:56:51.143137+0000","last_unstale":"2024-03-30T13:57:24.303839+0000","last_undegraded":"2024-03-30T13:57:24.303839+0000","last_fullsized":"2024-03-30T13:57:24.303839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:45:54.882185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.170386+0000","last_change":"2024-03-30T13:56:49.139846+0000","last_active":"2024-03-30T13:57:25.170386+0000","last_peered":"2024-03-30T13:57:25.170386+0000","last_clean":"2024-03-30T13:57:25.170386+0000","last_became_active":"2024-03-30T13:56:49.139589+0000","last_became_peered":"2024-03-30T13:56:49.139589+0000","last_unstale":"2024-03-30T13:57:25.170386+0000","last_undegraded":"2024-03-30T13:57:25.170386+0000","last_fullsized":"2024-03-30T13:57:25.170386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:37:22.628416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202543+0000","last_change":"2024-03-30T13:56:45.120873+0000","last_active":"2024-03-30T13:57:24.202543+0000","last_peered":"2024-03-30T13:57:24.202543+0000","last_clean":"2024-03-30T13:57:24.202543+0000","last_became_active":"2024-03-30T13:56:45.120226+0000","last_became_peered":"2024-03-30T13:56:45.120226+0000","last_unstale":"2024-03-30T13:57:24.202543+0000","last_undegraded":"2024-03-30T13:57:24.202543+0000","last_fullsized":"2024-03-30T13:57:24.202543+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:09:46.349553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612266+0000","last_change":"2024-03-30T13:56:47.128260+0000","last_active":"2024-03-30T13:57:24.612266+0000","last_peered":"2024-03-30T13:57:24.612266+0000","last_clean":"2024-03-30T13:57:24.612266+0000","last_became_active":"2024-03-30T13:56:47.127544+0000","last_became_peered":"2024-03-30T13:56:47.127544+0000","last_unstale":"2024-03-30T13:57:24.612266+0000","last_undegraded":"2024-03-30T13:57:24.612266+0000","last_fullsized":"2024-03-30T13:57:24.612266+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:14:37.168959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612503+0000","last_change":"2024-03-30T13:56:51.376808+0000","last_active":"2024-03-30T13:57:24.612503+0000","last_peered":"2024-03-30T13:57:24.612503+0000","last_clean":"2024-03-30T13:57:24.612503+0000","last_became_active":"2024-03-30T13:56:51.376489+0000","last_became_peered":"2024-03-30T13:56:51.376489+0000","last_unstale":"2024-03-30T13:57:24.612503+0000","last_undegraded":"2024-03-30T13:57:24.612503+0000","last_fullsized":"2024-03-30T13:57:24.612503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:32:42.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304104+0000","last_change":"2024-03-30T13:56:49.139874+0000","last_active":"2024-03-30T13:57:24.304104+0000","last_peered":"2024-03-30T13:57:24.304104+0000","last_clean":"2024-03-30T13:57:24.304104+0000","last_became_active":"2024-03-30T13:56:49.139320+0000","last_became_peered":"2024-03-30T13:56:49.139320+0000","last_unstale":"2024-03-30T13:57:24.304104+0000","last_undegraded":"2024-03-30T13:57:24.304104+0000","last_fullsized":"2024-03-30T13:57:24.304104+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:19:36.399488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291822+0000","last_change":"2024-03-30T13:56:45.117342+0000","last_active":"2024-03-30T13:57:24.291822+0000","last_peered":"2024-03-30T13:57:24.291822+0000","last_clean":"2024-03-30T13:57:24.291822+0000","last_became_active":"2024-03-30T13:56:45.116838+0000","last_became_peered":"2024-03-30T13:56:45.116838+0000","last_unstale":"2024-03-30T13:57:24.291822+0000","last_undegraded":"2024-03-30T13:57:24.291822+0000","last_fullsized":"2024-03-30T13:57:24.291822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:52:54.379597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612513+0000","last_change":"2024-03-30T13:56:48.037141+0000","last_active":"2024-03-30T13:57:24.612513+0000","last_peered":"2024-03-30T13:57:24.612513+0000","last_clean":"2024-03-30T13:57:24.612513+0000","last_became_active":"2024-03-30T13:56:48.036561+0000","last_became_peered":"2024-03-30T13:56:48.036561+0000","last_unstale":"2024-03-30T13:57:24.612513+0000","last_undegraded":"2024-03-30T13:57:24.612513+0000","last_fullsized":"2024-03-30T13:57:24.612513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:26:50.494560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.201918+0000","last_change":"2024-03-30T13:56:51.146984+0000","last_active":"2024-03-30T13:57:24.201918+0000","last_peered":"2024-03-30T13:57:24.201918+0000","last_clean":"2024-03-30T13:57:24.201918+0000","last_became_active":"2024-03-30T13:56:51.144901+0000","last_became_peered":"2024-03-30T13:56:51.144901+0000","last_unstale":"2024-03-30T13:57:24.201918+0000","last_undegraded":"2024-03-30T13:57:24.201918+0000","last_fullsized":"2024-03-30T13:57:24.201918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:17:05.923333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612799+0000","last_change":"2024-03-30T13:56:49.134564+0000","last_active":"2024-03-30T13:57:24.612799+0000","last_peered":"2024-03-30T13:57:24.612799+0000","last_clean":"2024-03-30T13:57:24.612799+0000","last_became_active":"2024-03-30T13:56:49.133687+0000","last_became_peered":"2024-03-30T13:56:49.133687+0000","last_unstale":"2024-03-30T13:57:24.612799+0000","last_undegraded":"2024-03-30T13:57:24.612799+0000","last_fullsized":"2024-03-30T13:57:24.612799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:17:20.063679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304328+0000","last_change":"2024-03-30T13:56:45.117684+0000","last_active":"2024-03-30T13:57:24.304328+0000","last_peered":"2024-03-30T13:57:24.304328+0000","last_clean":"2024-03-30T13:57:24.304328+0000","last_became_active":"2024-03-30T13:56:45.117377+0000","last_became_peered":"2024-03-30T13:56:45.117377+0000","last_unstale":"2024-03-30T13:57:24.304328+0000","last_undegraded":"2024-03-30T13:57:24.304328+0000","last_fullsized":"2024-03-30T13:57:24.304328+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:28:13.342179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204144+0000","last_change":"2024-03-30T13:56:47.129790+0000","last_active":"2024-03-30T13:57:24.204144+0000","last_peered":"2024-03-30T13:57:24.204144+0000","last_clean":"2024-03-30T13:57:24.204144+0000","last_became_active":"2024-03-30T13:56:47.129477+0000","last_became_peered":"2024-03-30T13:56:47.129477+0000","last_unstale":"2024-03-30T13:57:24.204144+0000","last_undegraded":"2024-03-30T13:57:24.204144+0000","last_fullsized":"2024-03-30T13:57:24.204144+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:00.772680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612284+0000","last_change":"2024-03-30T13:56:51.150812+0000","last_active":"2024-03-30T13:57:24.612284+0000","last_peered":"2024-03-30T13:57:24.612284+0000","last_clean":"2024-03-30T13:57:24.612284+0000","last_became_active":"2024-03-30T13:56:51.150460+0000","last_became_peered":"2024-03-30T13:56:51.150460+0000","last_unstale":"2024-03-30T13:57:24.612284+0000","last_undegraded":"2024-03-30T13:57:24.612284+0000","last_fullsized":"2024-03-30T13:57:24.612284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:48:49.857265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203922+0000","last_change":"2024-03-30T13:56:49.139114+0000","last_active":"2024-03-30T13:57:24.203922+0000","last_peered":"2024-03-30T13:57:24.203922+0000","last_clean":"2024-03-30T13:57:24.203922+0000","last_became_active":"2024-03-30T13:56:49.138750+0000","last_became_peered":"2024-03-30T13:56:49.138750+0000","last_unstale":"2024-03-30T13:57:24.203922+0000","last_undegraded":"2024-03-30T13:57:24.203922+0000","last_fullsized":"2024-03-30T13:57:24.203922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:18:22.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203678+0000","last_change":"2024-03-30T13:56:45.119524+0000","last_active":"2024-03-30T13:57:24.203678+0000","last_peered":"2024-03-30T13:57:24.203678+0000","last_clean":"2024-03-30T13:57:24.203678+0000","last_became_active":"2024-03-30T13:56:45.119230+0000","last_became_peered":"2024-03-30T13:56:45.119230+0000","last_unstale":"2024-03-30T13:57:24.203678+0000","last_undegraded":"2024-03-30T13:57:24.203678+0000","last_fullsized":"2024-03-30T13:57:24.203678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:43:22.784479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204472+0000","last_change":"2024-03-30T13:56:47.120553+0000","last_active":"2024-03-30T13:57:24.204472+0000","last_peered":"2024-03-30T13:57:24.204472+0000","last_clean":"2024-03-30T13:57:24.204472+0000","last_became_active":"2024-03-30T13:56:47.120189+0000","last_became_peered":"2024-03-30T13:56:47.120189+0000","last_unstale":"2024-03-30T13:57:24.204472+0000","last_undegraded":"2024-03-30T13:57:24.204472+0000","last_fullsized":"2024-03-30T13:57:24.204472+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:21:47.977524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202323+0000","last_change":"2024-03-30T13:56:51.151218+0000","last_active":"2024-03-30T13:57:24.202323+0000","last_peered":"2024-03-30T13:57:24.202323+0000","last_clean":"2024-03-30T13:57:24.202323+0000","last_became_active":"2024-03-30T13:56:51.150865+0000","last_became_peered":"2024-03-30T13:56:51.150865+0000","last_unstale":"2024-03-30T13:57:24.202323+0000","last_undegraded":"2024-03-30T13:57:24.202323+0000","last_fullsized":"2024-03-30T13:57:24.202323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:08:27.775166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.176054+0000","last_change":"2024-03-30T13:56:49.140625+0000","last_active":"2024-03-30T13:58:05.176054+0000","last_peered":"2024-03-30T13:58:05.176054+0000","last_clean":"2024-03-30T13:58:05.176054+0000","last_became_active":"2024-03-30T13:56:49.140340+0000","last_became_peered":"2024-03-30T13:56:49.140340+0000","last_unstale":"2024-03-30T13:58:05.176054+0000","last_undegraded":"2024-03-30T13:58:05.176054+0000","last_fullsized":"2024-03-30T13:58:05.176054+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:27:01.477794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303961+0000","last_change":"2024-03-30T13:56:45.120035+0000","last_active":"2024-03-30T13:57:24.303961+0000","last_peered":"2024-03-30T13:57:24.303961+0000","last_clean":"2024-03-30T13:57:24.303961+0000","last_became_active":"2024-03-30T13:56:45.119170+0000","last_became_peered":"2024-03-30T13:56:45.119170+0000","last_unstale":"2024-03-30T13:57:24.303961+0000","last_undegraded":"2024-03-30T13:57:24.303961+0000","last_fullsized":"2024-03-30T13:57:24.303961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:50:13.267995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":62,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612770+0000","last_change":"2024-03-30T13:57:09.184977+0000","last_active":"2024-03-30T13:57:24.612770+0000","last_peered":"2024-03-30T13:57:24.612770+0000","last_clean":"2024-03-30T13:57:24.612770+0000","last_became_active":"2024-03-30T13:57:09.173724+0000","last_became_peered":"2024-03-30T13:57:09.173724+0000","last_unstale":"2024-03-30T13:57:24.612770+0000","last_undegraded":"2024-03-30T13:57:24.612770+0000","last_fullsized":"2024-03-30T13:57:24.612770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:41:19.452412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204041+0000","last_change":"2024-03-30T13:56:51.146553+0000","last_active":"2024-03-30T13:57:24.204041+0000","last_peered":"2024-03-30T13:57:24.204041+0000","last_clean":"2024-03-30T13:57:24.204041+0000","last_became_active":"2024-03-30T13:56:51.146205+0000","last_became_peered":"2024-03-30T13:56:51.146205+0000","last_unstale":"2024-03-30T13:57:24.204041+0000","last_undegraded":"2024-03-30T13:57:24.204041+0000","last_fullsized":"2024-03-30T13:57:24.204041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:01:54.831679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.176308+0000","last_change":"2024-03-30T13:56:49.137965+0000","last_active":"2024-03-30T13:58:05.176308+0000","last_peered":"2024-03-30T13:58:05.176308+0000","last_clean":"2024-03-30T13:58:05.176308+0000","last_became_active":"2024-03-30T13:56:49.137672+0000","last_became_peered":"2024-03-30T13:56:49.137672+0000","last_unstale":"2024-03-30T13:58:05.176308+0000","last_undegraded":"2024-03-30T13:58:05.176308+0000","last_fullsized":"2024-03-30T13:58:05.176308+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:31:40.550492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202185+0000","last_change":"2024-03-30T13:56:45.120466+0000","last_active":"2024-03-30T13:57:24.202185+0000","last_peered":"2024-03-30T13:57:24.202185+0000","last_clean":"2024-03-30T13:57:24.202185+0000","last_became_active":"2024-03-30T13:56:45.118485+0000","last_became_peered":"2024-03-30T13:56:45.118485+0000","last_unstale":"2024-03-30T13:57:24.202185+0000","last_undegraded":"2024-03-30T13:57:24.202185+0000","last_fullsized":"2024-03-30T13:57:24.202185+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:20:48.928837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291687+0000","last_change":"2024-03-30T13:56:48.039279+0000","last_active":"2024-03-30T13:57:24.291687+0000","last_peered":"2024-03-30T13:57:24.291687+0000","last_clean":"2024-03-30T13:57:24.291687+0000","last_became_active":"2024-03-30T13:56:48.038872+0000","last_became_peered":"2024-03-30T13:56:48.038872+0000","last_unstale":"2024-03-30T13:57:24.291687+0000","last_undegraded":"2024-03-30T13:57:24.291687+0000","last_fullsized":"2024-03-30T13:57:24.291687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:26:27.204441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300365+0000","last_change":"2024-03-30T13:56:51.143070+0000","last_active":"2024-03-30T13:57:24.300365+0000","last_peered":"2024-03-30T13:57:24.300365+0000","last_clean":"2024-03-30T13:57:24.300365+0000","last_became_active":"2024-03-30T13:56:51.142763+0000","last_became_peered":"2024-03-30T13:56:51.142763+0000","last_unstale":"2024-03-30T13:57:24.300365+0000","last_undegraded":"2024-03-30T13:57:24.300365+0000","last_fullsized":"2024-03-30T13:57:24.300365+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:02:53.647876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613387+0000","last_change":"2024-03-30T13:56:49.139217+0000","last_active":"2024-03-30T13:57:24.613387+0000","last_peered":"2024-03-30T13:57:24.613387+0000","last_clean":"2024-03-30T13:57:24.613387+0000","last_became_active":"2024-03-30T13:56:49.138891+0000","last_became_peered":"2024-03-30T13:56:49.138891+0000","last_unstale":"2024-03-30T13:57:24.613387+0000","last_undegraded":"2024-03-30T13:57:24.613387+0000","last_fullsized":"2024-03-30T13:57:24.613387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:46:50.528300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304043+0000","last_change":"2024-03-30T13:56:45.116615+0000","last_active":"2024-03-30T13:57:24.304043+0000","last_peered":"2024-03-30T13:57:24.304043+0000","last_clean":"2024-03-30T13:57:24.304043+0000","last_became_active":"2024-03-30T13:56:45.116334+0000","last_became_peered":"2024-03-30T13:56:45.116334+0000","last_unstale":"2024-03-30T13:57:24.304043+0000","last_undegraded":"2024-03-30T13:57:24.304043+0000","last_fullsized":"2024-03-30T13:57:24.304043+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:30:13.716951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202717+0000","last_change":"2024-03-30T13:56:48.036822+0000","last_active":"2024-03-30T13:57:24.202717+0000","last_peered":"2024-03-30T13:57:24.202717+0000","last_clean":"2024-03-30T13:57:24.202717+0000","last_became_active":"2024-03-30T13:56:48.036471+0000","last_became_peered":"2024-03-30T13:56:48.036471+0000","last_unstale":"2024-03-30T13:57:24.202717+0000","last_undegraded":"2024-03-30T13:57:24.202717+0000","last_fullsized":"2024-03-30T13:57:24.202717+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:58:01.993816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304054+0000","last_change":"2024-03-30T13:56:51.376420+0000","last_active":"2024-03-30T13:57:24.304054+0000","last_peered":"2024-03-30T13:57:24.304054+0000","last_clean":"2024-03-30T13:57:24.304054+0000","last_became_active":"2024-03-30T13:56:51.375902+0000","last_became_peered":"2024-03-30T13:56:51.375902+0000","last_unstale":"2024-03-30T13:57:24.304054+0000","last_undegraded":"2024-03-30T13:57:24.304054+0000","last_fullsized":"2024-03-30T13:57:24.304054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:16:19.572835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291632+0000","last_change":"2024-03-30T13:56:49.143704+0000","last_active":"2024-03-30T13:57:24.291632+0000","last_peered":"2024-03-30T13:57:24.291632+0000","last_clean":"2024-03-30T13:57:24.291632+0000","last_became_active":"2024-03-30T13:56:49.143393+0000","last_became_peered":"2024-03-30T13:56:49.143393+0000","last_unstale":"2024-03-30T13:57:24.291632+0000","last_undegraded":"2024-03-30T13:57:24.291632+0000","last_fullsized":"2024-03-30T13:57:24.291632+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:51:23.903350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.169934+0000","last_change":"2024-03-30T13:56:45.122014+0000","last_active":"2024-03-30T13:57:25.169934+0000","last_peered":"2024-03-30T13:57:25.169934+0000","last_clean":"2024-03-30T13:57:25.169934+0000","last_became_active":"2024-03-30T13:56:45.121587+0000","last_became_peered":"2024-03-30T13:56:45.121587+0000","last_unstale":"2024-03-30T13:57:25.169934+0000","last_undegraded":"2024-03-30T13:57:25.169934+0000","last_fullsized":"2024-03-30T13:57:25.169934+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:02:50.921093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613476+0000","last_change":"2024-03-30T13:56:48.037393+0000","last_active":"2024-03-30T13:57:24.613476+0000","last_peered":"2024-03-30T13:57:24.613476+0000","last_clean":"2024-03-30T13:57:24.613476+0000","last_became_active":"2024-03-30T13:56:48.037084+0000","last_became_peered":"2024-03-30T13:56:48.037084+0000","last_unstale":"2024-03-30T13:57:24.613476+0000","last_undegraded":"2024-03-30T13:57:24.613476+0000","last_fullsized":"2024-03-30T13:57:24.613476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:38:43.687720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291704+0000","last_change":"2024-03-30T13:56:51.374229+0000","last_active":"2024-03-30T13:57:24.291704+0000","last_peered":"2024-03-30T13:57:24.291704+0000","last_clean":"2024-03-30T13:57:24.291704+0000","last_became_active":"2024-03-30T13:56:51.373906+0000","last_became_peered":"2024-03-30T13:56:51.373906+0000","last_unstale":"2024-03-30T13:57:24.291704+0000","last_undegraded":"2024-03-30T13:57:24.291704+0000","last_fullsized":"2024-03-30T13:57:24.291704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:45:25.296390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304335+0000","last_change":"2024-03-30T13:56:45.122862+0000","last_active":"2024-03-30T13:57:24.304335+0000","last_peered":"2024-03-30T13:57:24.304335+0000","last_clean":"2024-03-30T13:57:24.304335+0000","last_became_active":"2024-03-30T13:56:45.122609+0000","last_became_peered":"2024-03-30T13:56:45.122609+0000","last_unstale":"2024-03-30T13:57:24.304335+0000","last_undegraded":"2024-03-30T13:57:24.304335+0000","last_fullsized":"2024-03-30T13:57:24.304335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:33:11.217111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291156+0000","last_change":"2024-03-30T13:56:48.039492+0000","last_active":"2024-03-30T13:57:24.291156+0000","last_peered":"2024-03-30T13:57:24.291156+0000","last_clean":"2024-03-30T13:57:24.291156+0000","last_became_active":"2024-03-30T13:56:48.038898+0000","last_became_peered":"2024-03-30T13:56:48.038898+0000","last_unstale":"2024-03-30T13:57:24.291156+0000","last_undegraded":"2024-03-30T13:57:24.291156+0000","last_fullsized":"2024-03-30T13:57:24.291156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:52:14.536138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.176121+0000","last_change":"2024-03-30T13:56:49.140187+0000","last_active":"2024-03-30T13:58:05.176121+0000","last_peered":"2024-03-30T13:58:05.176121+0000","last_clean":"2024-03-30T13:58:05.176121+0000","last_became_active":"2024-03-30T13:56:49.139903+0000","last_became_peered":"2024-03-30T13:56:49.139903+0000","last_unstale":"2024-03-30T13:58:05.176121+0000","last_undegraded":"2024-03-30T13:58:05.176121+0000","last_fullsized":"2024-03-30T13:58:05.176121+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:34:22.725256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612610+0000","last_change":"2024-03-30T13:56:51.146611+0000","last_active":"2024-03-30T13:57:24.612610+0000","last_peered":"2024-03-30T13:57:24.612610+0000","last_clean":"2024-03-30T13:57:24.612610+0000","last_became_active":"2024-03-30T13:56:51.146277+0000","last_became_peered":"2024-03-30T13:56:51.146277+0000","last_unstale":"2024-03-30T13:57:24.612610+0000","last_undegraded":"2024-03-30T13:57:24.612610+0000","last_fullsized":"2024-03-30T13:57:24.612610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:08:12.492187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304360+0000","last_change":"2024-03-30T13:56:45.118302+0000","last_active":"2024-03-30T13:57:24.304360+0000","last_peered":"2024-03-30T13:57:24.304360+0000","last_clean":"2024-03-30T13:57:24.304360+0000","last_became_active":"2024-03-30T13:56:45.117976+0000","last_became_peered":"2024-03-30T13:56:45.117976+0000","last_unstale":"2024-03-30T13:57:24.304360+0000","last_undegraded":"2024-03-30T13:57:24.304360+0000","last_fullsized":"2024-03-30T13:57:24.304360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:07:02.148184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203408+0000","last_change":"2024-03-30T13:57:09.179801+0000","last_active":"2024-03-30T13:57:24.203408+0000","last_peered":"2024-03-30T13:57:24.203408+0000","last_clean":"2024-03-30T13:57:24.203408+0000","last_became_active":"2024-03-30T13:57:09.173514+0000","last_became_peered":"2024-03-30T13:57:09.173514+0000","last_unstale":"2024-03-30T13:57:24.203408+0000","last_undegraded":"2024-03-30T13:57:24.203408+0000","last_fullsized":"2024-03-30T13:57:24.203408+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:54:56.001721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.299618+0000","last_change":"2024-03-30T13:56:49.134000+0000","last_active":"2024-03-30T13:57:24.299618+0000","last_peered":"2024-03-30T13:57:24.299618+0000","last_clean":"2024-03-30T13:57:24.299618+0000","last_became_active":"2024-03-30T13:56:49.133135+0000","last_became_peered":"2024-03-30T13:56:49.133135+0000","last_unstale":"2024-03-30T13:57:24.299618+0000","last_undegraded":"2024-03-30T13:57:24.299618+0000","last_fullsized":"2024-03-30T13:57:24.299618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:04:33.457010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303652+0000","last_change":"2024-03-30T13:56:51.147026+0000","last_active":"2024-03-30T13:57:24.303652+0000","last_peered":"2024-03-30T13:57:24.303652+0000","last_clean":"2024-03-30T13:57:24.303652+0000","last_became_active":"2024-03-30T13:56:51.146720+0000","last_became_peered":"2024-03-30T13:56:51.146720+0000","last_unstale":"2024-03-30T13:57:24.303652+0000","last_undegraded":"2024-03-30T13:57:24.303652+0000","last_fullsized":"2024-03-30T13:57:24.303652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:55:05.576387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.299958+0000","last_change":"2024-03-30T13:56:49.137678+0000","last_active":"2024-03-30T13:57:24.299958+0000","last_peered":"2024-03-30T13:57:24.299958+0000","last_clean":"2024-03-30T13:57:24.299958+0000","last_became_active":"2024-03-30T13:56:49.137366+0000","last_became_peered":"2024-03-30T13:56:49.137366+0000","last_unstale":"2024-03-30T13:57:24.299958+0000","last_undegraded":"2024-03-30T13:57:24.299958+0000","last_fullsized":"2024-03-30T13:57:24.299958+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:05:57.360946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304500+0000","last_change":"2024-03-30T13:56:45.120209+0000","last_active":"2024-03-30T13:57:24.304500+0000","last_peered":"2024-03-30T13:57:24.304500+0000","last_clean":"2024-03-30T13:57:24.304500+0000","last_became_active":"2024-03-30T13:56:45.119107+0000","last_became_peered":"2024-03-30T13:56:45.119107+0000","last_unstale":"2024-03-30T13:57:24.304500+0000","last_undegraded":"2024-03-30T13:57:24.304500+0000","last_fullsized":"2024-03-30T13:57:24.304500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:35:12.748914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.170147+0000","last_change":"2024-03-30T13:57:09.151659+0000","last_active":"2024-03-30T13:57:25.170147+0000","last_peered":"2024-03-30T13:57:25.170147+0000","last_clean":"2024-03-30T13:57:25.170147+0000","last_became_active":"2024-03-30T13:57:09.141903+0000","last_became_peered":"2024-03-30T13:57:09.141903+0000","last_unstale":"2024-03-30T13:57:25.170147+0000","last_undegraded":"2024-03-30T13:57:25.170147+0000","last_fullsized":"2024-03-30T13:57:25.170147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:10:57.530537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.201980+0000","last_change":"2024-03-30T13:56:51.147843+0000","last_active":"2024-03-30T13:57:24.201980+0000","last_peered":"2024-03-30T13:57:24.201980+0000","last_clean":"2024-03-30T13:57:24.201980+0000","last_became_active":"2024-03-30T13:56:51.147505+0000","last_became_peered":"2024-03-30T13:56:51.147505+0000","last_unstale":"2024-03-30T13:57:24.201980+0000","last_undegraded":"2024-03-30T13:57:24.201980+0000","last_fullsized":"2024-03-30T13:57:24.201980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:22:39.554642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300002+0000","last_change":"2024-03-30T13:56:49.140253+0000","last_active":"2024-03-30T13:57:24.300002+0000","last_peered":"2024-03-30T13:57:24.300002+0000","last_clean":"2024-03-30T13:57:24.300002+0000","last_became_active":"2024-03-30T13:56:49.139900+0000","last_became_peered":"2024-03-30T13:56:49.139900+0000","last_unstale":"2024-03-30T13:57:24.300002+0000","last_undegraded":"2024-03-30T13:57:24.300002+0000","last_fullsized":"2024-03-30T13:57:24.300002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:05:17.865193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291618+0000","last_change":"2024-03-30T13:56:45.119791+0000","last_active":"2024-03-30T13:57:24.291618+0000","last_peered":"2024-03-30T13:57:24.291618+0000","last_clean":"2024-03-30T13:57:24.291618+0000","last_became_active":"2024-03-30T13:56:45.119497+0000","last_became_peered":"2024-03-30T13:56:45.119497+0000","last_unstale":"2024-03-30T13:57:24.291618+0000","last_undegraded":"2024-03-30T13:57:24.291618+0000","last_fullsized":"2024-03-30T13:57:24.291618+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:27:07.256388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203503+0000","last_change":"2024-03-30T13:56:47.120630+0000","last_active":"2024-03-30T13:57:24.203503+0000","last_peered":"2024-03-30T13:57:24.203503+0000","last_clean":"2024-03-30T13:57:24.203503+0000","last_became_active":"2024-03-30T13:56:47.120245+0000","last_became_peered":"2024-03-30T13:56:47.120245+0000","last_unstale":"2024-03-30T13:57:24.203503+0000","last_undegraded":"2024-03-30T13:57:24.203503+0000","last_fullsized":"2024-03-30T13:57:24.203503+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:13:52.849073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613157+0000","last_change":"2024-03-30T13:56:51.144198+0000","last_active":"2024-03-30T13:57:24.613157+0000","last_peered":"2024-03-30T13:57:24.613157+0000","last_clean":"2024-03-30T13:57:24.613157+0000","last_became_active":"2024-03-30T13:56:51.143783+0000","last_became_peered":"2024-03-30T13:56:51.143783+0000","last_unstale":"2024-03-30T13:57:24.613157+0000","last_undegraded":"2024-03-30T13:57:24.613157+0000","last_fullsized":"2024-03-30T13:57:24.613157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:09:38.179728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291210+0000","last_change":"2024-03-30T13:56:48.120221+0000","last_active":"2024-03-30T13:57:24.291210+0000","last_peered":"2024-03-30T13:57:24.291210+0000","last_clean":"2024-03-30T13:57:24.291210+0000","last_became_active":"2024-03-30T13:56:44.113778+0000","last_became_peered":"2024-03-30T13:56:44.113778+0000","last_unstale":"2024-03-30T13:57:24.291210+0000","last_undegraded":"2024-03-30T13:57:24.291210+0000","last_fullsized":"2024-03-30T13:57:24.291210+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_clean_scrub_stamp":"2024-03-30T13:56:43.098310+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:48:43.293005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051844900000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203303+0000","last_change":"2024-03-30T13:56:45.120368+0000","last_active":"2024-03-30T13:57:24.203303+0000","last_peered":"2024-03-30T13:57:24.203303+0000","last_clean":"2024-03-30T13:57:24.203303+0000","last_became_active":"2024-03-30T13:56:45.120077+0000","last_became_peered":"2024-03-30T13:56:45.120077+0000","last_unstale":"2024-03-30T13:57:24.203303+0000","last_undegraded":"2024-03-30T13:57:24.203303+0000","last_fullsized":"2024-03-30T13:57:24.203303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:22:03.966315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":95,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.175961+0000","last_change":"2024-03-30T13:56:47.131477+0000","last_active":"2024-03-30T13:58:05.175961+0000","last_peered":"2024-03-30T13:58:05.175961+0000","last_clean":"2024-03-30T13:58:05.175961+0000","last_became_active":"2024-03-30T13:56:47.131044+0000","last_became_peered":"2024-03-30T13:56:47.131044+0000","last_unstale":"2024-03-30T13:58:05.175961+0000","last_undegraded":"2024-03-30T13:58:05.175961+0000","last_fullsized":"2024-03-30T13:58:05.175961+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:48:16.154473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202317+0000","last_change":"2024-03-30T13:56:49.130328+0000","last_active":"2024-03-30T13:57:24.202317+0000","last_peered":"2024-03-30T13:57:24.202317+0000","last_clean":"2024-03-30T13:57:24.202317+0000","last_became_active":"2024-03-30T13:56:49.130031+0000","last_became_peered":"2024-03-30T13:56:49.130031+0000","last_unstale":"2024-03-30T13:57:24.202317+0000","last_undegraded":"2024-03-30T13:57:24.202317+0000","last_fullsized":"2024-03-30T13:57:24.202317+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:36:51.024682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304554+0000","last_change":"2024-03-30T13:56:51.374278+0000","last_active":"2024-03-30T13:57:24.304554+0000","last_peered":"2024-03-30T13:57:24.304554+0000","last_clean":"2024-03-30T13:57:24.304554+0000","last_became_active":"2024-03-30T13:56:51.373977+0000","last_became_peered":"2024-03-30T13:56:51.373977+0000","last_unstale":"2024-03-30T13:57:24.304554+0000","last_undegraded":"2024-03-30T13:57:24.304554+0000","last_fullsized":"2024-03-30T13:57:24.304554+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:24:58.288537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203105+0000","last_change":"2024-03-30T13:56:45.116511+0000","last_active":"2024-03-30T13:57:24.203105+0000","last_peered":"2024-03-30T13:57:24.203105+0000","last_clean":"2024-03-30T13:57:24.203105+0000","last_became_active":"2024-03-30T13:56:45.116189+0000","last_became_peered":"2024-03-30T13:56:45.116189+0000","last_unstale":"2024-03-30T13:57:24.203105+0000","last_undegraded":"2024-03-30T13:57:24.203105+0000","last_fullsized":"2024-03-30T13:57:24.203105+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:06:00.495646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.299765+0000","last_change":"2024-03-30T13:56:48.119693+0000","last_active":"2024-03-30T13:57:24.299765+0000","last_peered":"2024-03-30T13:57:24.299765+0000","last_clean":"2024-03-30T13:57:24.299765+0000","last_became_active":"2024-03-30T13:56:44.115568+0000","last_became_peered":"2024-03-30T13:56:44.115568+0000","last_unstale":"2024-03-30T13:57:24.299765+0000","last_undegraded":"2024-03-30T13:57:24.299765+0000","last_fullsized":"2024-03-30T13:57:24.299765+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_clean_scrub_stamp":"2024-03-30T13:56:43.098310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:01:25.483974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001808111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205747+0000","last_change":"2024-03-30T13:56:47.124567+0000","last_active":"2024-03-30T13:57:24.205747+0000","last_peered":"2024-03-30T13:57:24.205747+0000","last_clean":"2024-03-30T13:57:24.205747+0000","last_became_active":"2024-03-30T13:56:47.124221+0000","last_became_peered":"2024-03-30T13:56:47.124221+0000","last_unstale":"2024-03-30T13:57:24.205747+0000","last_undegraded":"2024-03-30T13:57:24.205747+0000","last_fullsized":"2024-03-30T13:57:24.205747+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:28:29.960992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300359+0000","last_change":"2024-03-30T13:56:49.138374+0000","last_active":"2024-03-30T13:57:24.300359+0000","last_peered":"2024-03-30T13:57:24.300359+0000","last_clean":"2024-03-30T13:57:24.300359+0000","last_became_active":"2024-03-30T13:56:49.137770+0000","last_became_peered":"2024-03-30T13:56:49.137770+0000","last_unstale":"2024-03-30T13:57:24.300359+0000","last_undegraded":"2024-03-30T13:57:24.300359+0000","last_fullsized":"2024-03-30T13:57:24.300359+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:48:05.776781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291484+0000","last_change":"2024-03-30T13:56:51.145093+0000","last_active":"2024-03-30T13:57:24.291484+0000","last_peered":"2024-03-30T13:57:24.291484+0000","last_clean":"2024-03-30T13:57:24.291484+0000","last_became_active":"2024-03-30T13:56:51.144742+0000","last_became_peered":"2024-03-30T13:56:51.144742+0000","last_unstale":"2024-03-30T13:57:24.291484+0000","last_undegraded":"2024-03-30T13:57:24.291484+0000","last_fullsized":"2024-03-30T13:57:24.291484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:01:03.127914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:26.294665+0000","last_change":"2024-03-30T13:56:31.734197+0000","last_active":"2024-03-30T13:57:26.294665+0000","last_peered":"2024-03-30T13:57:26.294665+0000","last_clean":"2024-03-30T13:57:26.294665+0000","last_became_active":"2024-03-30T13:56:31.733805+0000","last_became_peered":"2024-03-30T13:56:31.733805+0000","last_unstale":"2024-03-30T13:57:26.294665+0000","last_undegraded":"2024-03-30T13:57:26.294665+0000","last_fullsized":"2024-03-30T13:57:26.294665+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:54:25.686692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:54:25.686692+0000","last_clean_scrub_stamp":"2024-03-30T13:54:25.686692+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:13:07.005289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612740+0000","last_change":"2024-03-30T13:56:45.121763+0000","last_active":"2024-03-30T13:57:24.612740+0000","last_peered":"2024-03-30T13:57:24.612740+0000","last_clean":"2024-03-30T13:57:24.612740+0000","last_became_active":"2024-03-30T13:56:45.121424+0000","last_became_peered":"2024-03-30T13:56:45.121424+0000","last_unstale":"2024-03-30T13:57:24.612740+0000","last_undegraded":"2024-03-30T13:57:24.612740+0000","last_fullsized":"2024-03-30T13:57:24.612740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:29:50.114112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":69,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.176100+0000","last_change":"2024-03-30T13:56:48.038014+0000","last_active":"2024-03-30T13:58:05.176100+0000","last_peered":"2024-03-30T13:58:05.176100+0000","last_clean":"2024-03-30T13:58:05.176100+0000","last_became_active":"2024-03-30T13:56:48.037609+0000","last_became_peered":"2024-03-30T13:56:48.037609+0000","last_unstale":"2024-03-30T13:58:05.176100+0000","last_undegraded":"2024-03-30T13:58:05.176100+0000","last_fullsized":"2024-03-30T13:58:05.176100+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:46:30.568119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.304199+0000","last_change":"2024-03-30T13:56:49.137841+0000","last_active":"2024-03-30T13:57:24.304199+0000","last_peered":"2024-03-30T13:57:24.304199+0000","last_clean":"2024-03-30T13:57:24.304199+0000","last_became_active":"2024-03-30T13:56:49.137576+0000","last_became_peered":"2024-03-30T13:56:49.137576+0000","last_unstale":"2024-03-30T13:57:24.304199+0000","last_undegraded":"2024-03-30T13:57:24.304199+0000","last_fullsized":"2024-03-30T13:57:24.304199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:03:10.485823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204549+0000","last_change":"2024-03-30T13:56:51.148056+0000","last_active":"2024-03-30T13:57:24.204549+0000","last_peered":"2024-03-30T13:57:24.204549+0000","last_clean":"2024-03-30T13:57:24.204549+0000","last_became_active":"2024-03-30T13:56:51.147616+0000","last_became_peered":"2024-03-30T13:56:51.147616+0000","last_unstale":"2024-03-30T13:57:24.204549+0000","last_undegraded":"2024-03-30T13:57:24.204549+0000","last_fullsized":"2024-03-30T13:57:24.204549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:44:43.140295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202156+0000","last_change":"2024-03-30T13:56:45.116207+0000","last_active":"2024-03-30T13:57:24.202156+0000","last_peered":"2024-03-30T13:57:24.202156+0000","last_clean":"2024-03-30T13:57:24.202156+0000","last_became_active":"2024-03-30T13:56:45.115723+0000","last_became_peered":"2024-03-30T13:56:45.115723+0000","last_unstale":"2024-03-30T13:57:24.202156+0000","last_undegraded":"2024-03-30T13:57:24.202156+0000","last_fullsized":"2024-03-30T13:57:24.202156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:28:05.722637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":123,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:09.165451+0000","last_change":"2024-03-30T13:56:48.120793+0000","last_active":"2024-03-30T13:58:09.165451+0000","last_peered":"2024-03-30T13:58:09.165451+0000","last_clean":"2024-03-30T13:58:09.165451+0000","last_became_active":"2024-03-30T13:56:44.117022+0000","last_became_peered":"2024-03-30T13:56:44.117022+0000","last_unstale":"2024-03-30T13:58:09.165451+0000","last_undegraded":"2024-03-30T13:58:09.165451+0000","last_fullsized":"2024-03-30T13:58:09.165451+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:43.098310+0000","last_clean_scrub_stamp":"2024-03-30T13:56:43.098310+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:37:47.279259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093502400000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202481+0000","last_change":"2024-03-30T13:56:47.124453+0000","last_active":"2024-03-30T13:57:24.202481+0000","last_peered":"2024-03-30T13:57:24.202481+0000","last_clean":"2024-03-30T13:57:24.202481+0000","last_became_active":"2024-03-30T13:56:47.124164+0000","last_became_peered":"2024-03-30T13:56:47.124164+0000","last_unstale":"2024-03-30T13:57:24.202481+0000","last_undegraded":"2024-03-30T13:57:24.202481+0000","last_fullsized":"2024-03-30T13:57:24.202481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:27:29.282192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291756+0000","last_change":"2024-03-30T13:56:49.141989+0000","last_active":"2024-03-30T13:57:24.291756+0000","last_peered":"2024-03-30T13:57:24.291756+0000","last_clean":"2024-03-30T13:57:24.291756+0000","last_became_active":"2024-03-30T13:56:49.141714+0000","last_became_peered":"2024-03-30T13:56:49.141714+0000","last_unstale":"2024-03-30T13:57:24.291756+0000","last_undegraded":"2024-03-30T13:57:24.291756+0000","last_fullsized":"2024-03-30T13:57:24.291756+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:58:01.263979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202767+0000","last_change":"2024-03-30T13:56:51.376844+0000","last_active":"2024-03-30T13:57:24.202767+0000","last_peered":"2024-03-30T13:57:24.202767+0000","last_clean":"2024-03-30T13:57:24.202767+0000","last_became_active":"2024-03-30T13:56:51.376490+0000","last_became_peered":"2024-03-30T13:56:51.376490+0000","last_unstale":"2024-03-30T13:57:24.202767+0000","last_undegraded":"2024-03-30T13:57:24.202767+0000","last_fullsized":"2024-03-30T13:57:24.202767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:05:39.060308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204754+0000","last_change":"2024-03-30T13:56:45.116114+0000","last_active":"2024-03-30T13:57:24.204754+0000","last_peered":"2024-03-30T13:57:24.204754+0000","last_clean":"2024-03-30T13:57:24.204754+0000","last_became_active":"2024-03-30T13:56:45.115817+0000","last_became_peered":"2024-03-30T13:56:45.115817+0000","last_unstale":"2024-03-30T13:57:24.204754+0000","last_undegraded":"2024-03-30T13:57:24.204754+0000","last_fullsized":"2024-03-30T13:57:24.204754+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:02:48.206255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203308+0000","last_change":"2024-03-30T13:56:48.036836+0000","last_active":"2024-03-30T13:57:24.203308+0000","last_peered":"2024-03-30T13:57:24.203308+0000","last_clean":"2024-03-30T13:57:24.203308+0000","last_became_active":"2024-03-30T13:56:48.036484+0000","last_became_peered":"2024-03-30T13:56:48.036484+0000","last_unstale":"2024-03-30T13:57:24.203308+0000","last_undegraded":"2024-03-30T13:57:24.203308+0000","last_fullsized":"2024-03-30T13:57:24.203308+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T01:06:31.777370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.170753+0000","last_change":"2024-03-30T13:56:49.138264+0000","last_active":"2024-03-30T13:57:25.170753+0000","last_peered":"2024-03-30T13:57:25.170753+0000","last_clean":"2024-03-30T13:57:25.170753+0000","last_became_active":"2024-03-30T13:56:49.137954+0000","last_became_peered":"2024-03-30T13:56:49.137954+0000","last_unstale":"2024-03-30T13:57:25.170753+0000","last_undegraded":"2024-03-30T13:57:25.170753+0000","last_fullsized":"2024-03-30T13:57:25.170753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:31:01.533147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.201918+0000","last_change":"2024-03-30T13:56:51.149024+0000","last_active":"2024-03-30T13:57:24.201918+0000","last_peered":"2024-03-30T13:57:24.201918+0000","last_clean":"2024-03-30T13:57:24.201918+0000","last_became_active":"2024-03-30T13:56:51.148658+0000","last_became_peered":"2024-03-30T13:56:51.148658+0000","last_unstale":"2024-03-30T13:57:24.201918+0000","last_undegraded":"2024-03-30T13:57:24.201918+0000","last_fullsized":"2024-03-30T13:57:24.201918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:56:49.675264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291525+0000","last_change":"2024-03-30T13:56:45.116248+0000","last_active":"2024-03-30T13:57:24.291525+0000","last_peered":"2024-03-30T13:57:24.291525+0000","last_clean":"2024-03-30T13:57:24.291525+0000","last_became_active":"2024-03-30T13:56:45.115966+0000","last_became_peered":"2024-03-30T13:56:45.115966+0000","last_unstale":"2024-03-30T13:57:24.291525+0000","last_undegraded":"2024-03-30T13:57:24.291525+0000","last_fullsized":"2024-03-30T13:57:24.291525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T23:35:48.959141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204025+0000","last_change":"2024-03-30T13:56:47.125784+0000","last_active":"2024-03-30T13:57:24.204025+0000","last_peered":"2024-03-30T13:57:24.204025+0000","last_clean":"2024-03-30T13:57:24.204025+0000","last_became_active":"2024-03-30T13:56:47.125437+0000","last_became_peered":"2024-03-30T13:56:47.125437+0000","last_unstale":"2024-03-30T13:57:24.204025+0000","last_undegraded":"2024-03-30T13:57:24.204025+0000","last_fullsized":"2024-03-30T13:57:24.204025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:16:32.723025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.175899+0000","last_change":"2024-03-30T13:56:49.136739+0000","last_active":"2024-03-30T13:58:05.175899+0000","last_peered":"2024-03-30T13:58:05.175899+0000","last_clean":"2024-03-30T13:58:05.175899+0000","last_became_active":"2024-03-30T13:56:49.136450+0000","last_became_peered":"2024-03-30T13:56:49.136450+0000","last_unstale":"2024-03-30T13:58:05.175899+0000","last_undegraded":"2024-03-30T13:58:05.175899+0000","last_fullsized":"2024-03-30T13:58:05.175899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:35:40.590636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.303831+0000","last_change":"2024-03-30T13:56:51.376269+0000","last_active":"2024-03-30T13:57:24.303831+0000","last_peered":"2024-03-30T13:57:24.303831+0000","last_clean":"2024-03-30T13:57:24.303831+0000","last_became_active":"2024-03-30T13:56:51.375831+0000","last_became_peered":"2024-03-30T13:56:51.375831+0000","last_unstale":"2024-03-30T13:57:24.303831+0000","last_undegraded":"2024-03-30T13:57:24.303831+0000","last_fullsized":"2024-03-30T13:57:24.303831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:44.828337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.202614+0000","last_change":"2024-03-30T13:56:45.115397+0000","last_active":"2024-03-30T13:57:24.202614+0000","last_peered":"2024-03-30T13:57:24.202614+0000","last_clean":"2024-03-30T13:57:24.202614+0000","last_became_active":"2024-03-30T13:56:45.115143+0000","last_became_peered":"2024-03-30T13:56:45.115143+0000","last_unstale":"2024-03-30T13:57:24.202614+0000","last_undegraded":"2024-03-30T13:57:24.202614+0000","last_fullsized":"2024-03-30T13:57:24.202614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:08:22.202665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203385+0000","last_change":"2024-03-30T13:56:47.118382+0000","last_active":"2024-03-30T13:57:24.203385+0000","last_peered":"2024-03-30T13:57:24.203385+0000","last_clean":"2024-03-30T13:57:24.203385+0000","last_became_active":"2024-03-30T13:56:47.117854+0000","last_became_peered":"2024-03-30T13:56:47.117854+0000","last_unstale":"2024-03-30T13:57:24.203385+0000","last_undegraded":"2024-03-30T13:57:24.203385+0000","last_fullsized":"2024-03-30T13:57:24.203385+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:33:09.207059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.176255+0000","last_change":"2024-03-30T13:56:49.138280+0000","last_active":"2024-03-30T13:58:05.176255+0000","last_peered":"2024-03-30T13:58:05.176255+0000","last_clean":"2024-03-30T13:58:05.176255+0000","last_became_active":"2024-03-30T13:56:49.137991+0000","last_became_peered":"2024-03-30T13:56:49.137991+0000","last_unstale":"2024-03-30T13:58:05.176255+0000","last_undegraded":"2024-03-30T13:58:05.176255+0000","last_fullsized":"2024-03-30T13:58:05.176255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:53:27.179411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203226+0000","last_change":"2024-03-30T13:56:51.146969+0000","last_active":"2024-03-30T13:57:24.203226+0000","last_peered":"2024-03-30T13:57:24.203226+0000","last_clean":"2024-03-30T13:57:24.203226+0000","last_became_active":"2024-03-30T13:56:51.146674+0000","last_became_peered":"2024-03-30T13:56:51.146674+0000","last_unstale":"2024-03-30T13:57:24.203226+0000","last_undegraded":"2024-03-30T13:57:24.203226+0000","last_fullsized":"2024-03-30T13:57:24.203226+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T22:12:14.805690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.613279+0000","last_change":"2024-03-30T13:56:45.119767+0000","last_active":"2024-03-30T13:57:24.613279+0000","last_peered":"2024-03-30T13:57:24.613279+0000","last_clean":"2024-03-30T13:57:24.613279+0000","last_became_active":"2024-03-30T13:56:45.119379+0000","last_became_peered":"2024-03-30T13:56:45.119379+0000","last_unstale":"2024-03-30T13:57:24.613279+0000","last_undegraded":"2024-03-30T13:57:24.613279+0000","last_fullsized":"2024-03-30T13:57:24.613279+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:41.829117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612513+0000","last_change":"2024-03-30T13:56:48.039657+0000","last_active":"2024-03-30T13:57:24.612513+0000","last_peered":"2024-03-30T13:57:24.612513+0000","last_clean":"2024-03-30T13:57:24.612513+0000","last_became_active":"2024-03-30T13:56:48.039120+0000","last_became_peered":"2024-03-30T13:56:48.039120+0000","last_unstale":"2024-03-30T13:57:24.612513+0000","last_undegraded":"2024-03-30T13:57:24.612513+0000","last_fullsized":"2024-03-30T13:57:24.612513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:16:16.597222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203150+0000","last_change":"2024-03-30T13:56:49.135107+0000","last_active":"2024-03-30T13:57:24.203150+0000","last_peered":"2024-03-30T13:57:24.203150+0000","last_clean":"2024-03-30T13:57:24.203150+0000","last_became_active":"2024-03-30T13:56:49.133957+0000","last_became_peered":"2024-03-30T13:56:49.133957+0000","last_unstale":"2024-03-30T13:57:24.203150+0000","last_undegraded":"2024-03-30T13:57:24.203150+0000","last_fullsized":"2024-03-30T13:57:24.203150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:43:55.232949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291361+0000","last_change":"2024-03-30T13:56:51.144275+0000","last_active":"2024-03-30T13:57:24.291361+0000","last_peered":"2024-03-30T13:57:24.291361+0000","last_clean":"2024-03-30T13:57:24.291361+0000","last_became_active":"2024-03-30T13:56:51.143965+0000","last_became_peered":"2024-03-30T13:56:51.143965+0000","last_unstale":"2024-03-30T13:57:24.291361+0000","last_undegraded":"2024-03-30T13:57:24.291361+0000","last_fullsized":"2024-03-30T13:57:24.291361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:25:23.436503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.176243+0000","last_change":"2024-03-30T13:56:49.137904+0000","last_active":"2024-03-30T13:58:05.176243+0000","last_peered":"2024-03-30T13:58:05.176243+0000","last_clean":"2024-03-30T13:58:05.176243+0000","last_became_active":"2024-03-30T13:56:49.137627+0000","last_became_peered":"2024-03-30T13:56:49.137627+0000","last_unstale":"2024-03-30T13:58:05.176243+0000","last_undegraded":"2024-03-30T13:58:05.176243+0000","last_fullsized":"2024-03-30T13:58:05.176243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:53:23.470116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612402+0000","last_change":"2024-03-30T13:56:45.118344+0000","last_active":"2024-03-30T13:57:24.612402+0000","last_peered":"2024-03-30T13:57:24.612402+0000","last_clean":"2024-03-30T13:57:24.612402+0000","last_became_active":"2024-03-30T13:56:45.117633+0000","last_became_peered":"2024-03-30T13:56:45.117633+0000","last_unstale":"2024-03-30T13:57:24.612402+0000","last_undegraded":"2024-03-30T13:57:24.612402+0000","last_fullsized":"2024-03-30T13:57:24.612402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:57:19.902679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.205806+0000","last_change":"2024-03-30T13:56:47.127609+0000","last_active":"2024-03-30T13:57:24.205806+0000","last_peered":"2024-03-30T13:57:24.205806+0000","last_clean":"2024-03-30T13:57:24.205806+0000","last_became_active":"2024-03-30T13:56:47.126134+0000","last_became_peered":"2024-03-30T13:56:47.126134+0000","last_unstale":"2024-03-30T13:57:24.205806+0000","last_undegraded":"2024-03-30T13:57:24.205806+0000","last_fullsized":"2024-03-30T13:57:24.205806+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:53:09.400906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612581+0000","last_change":"2024-03-30T13:56:51.374114+0000","last_active":"2024-03-30T13:57:24.612581+0000","last_peered":"2024-03-30T13:57:24.612581+0000","last_clean":"2024-03-30T13:57:24.612581+0000","last_became_active":"2024-03-30T13:56:51.373775+0000","last_became_peered":"2024-03-30T13:56:51.373775+0000","last_unstale":"2024-03-30T13:57:24.612581+0000","last_undegraded":"2024-03-30T13:57:24.612581+0000","last_fullsized":"2024-03-30T13:57:24.612581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T00:32:18.819817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.291692+0000","last_change":"2024-03-30T13:56:49.142752+0000","last_active":"2024-03-30T13:57:24.291692+0000","last_peered":"2024-03-30T13:57:24.291692+0000","last_clean":"2024-03-30T13:57:24.291692+0000","last_became_active":"2024-03-30T13:56:49.142488+0000","last_became_peered":"2024-03-30T13:56:49.142488+0000","last_unstale":"2024-03-30T13:57:24.291692+0000","last_undegraded":"2024-03-30T13:57:24.291692+0000","last_fullsized":"2024-03-30T13:57:24.291692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:07:26.493606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203256+0000","last_change":"2024-03-30T13:56:45.116294+0000","last_active":"2024-03-30T13:57:24.203256+0000","last_peered":"2024-03-30T13:57:24.203256+0000","last_clean":"2024-03-30T13:57:24.203256+0000","last_became_active":"2024-03-30T13:56:45.115982+0000","last_became_peered":"2024-03-30T13:56:45.115982+0000","last_unstale":"2024-03-30T13:57:24.203256+0000","last_undegraded":"2024-03-30T13:57:24.203256+0000","last_fullsized":"2024-03-30T13:57:24.203256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:26:49.056274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203866+0000","last_change":"2024-03-30T13:56:47.121770+0000","last_active":"2024-03-30T13:57:24.203866+0000","last_peered":"2024-03-30T13:57:24.203866+0000","last_clean":"2024-03-30T13:57:24.203866+0000","last_became_active":"2024-03-30T13:56:47.121315+0000","last_became_peered":"2024-03-30T13:56:47.121315+0000","last_unstale":"2024-03-30T13:57:24.203866+0000","last_undegraded":"2024-03-30T13:57:24.203866+0000","last_fullsized":"2024-03-30T13:57:24.203866+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:42:17.898358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.300677+0000","last_change":"2024-03-30T13:56:51.144168+0000","last_active":"2024-03-30T13:57:24.300677+0000","last_peered":"2024-03-30T13:57:24.300677+0000","last_clean":"2024-03-30T13:57:24.300677+0000","last_became_active":"2024-03-30T13:56:51.143711+0000","last_became_peered":"2024-03-30T13:56:51.143711+0000","last_unstale":"2024-03-30T13:57:24.300677+0000","last_undegraded":"2024-03-30T13:57:24.300677+0000","last_fullsized":"2024-03-30T13:57:24.300677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:57:23.216165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204990+0000","last_change":"2024-03-30T13:56:49.129026+0000","last_active":"2024-03-30T13:57:24.204990+0000","last_peered":"2024-03-30T13:57:24.204990+0000","last_clean":"2024-03-30T13:57:24.204990+0000","last_became_active":"2024-03-30T13:56:49.128710+0000","last_became_peered":"2024-03-30T13:56:49.128710+0000","last_unstale":"2024-03-30T13:57:24.204990+0000","last_undegraded":"2024-03-30T13:57:24.204990+0000","last_fullsized":"2024-03-30T13:57:24.204990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:09:59.892110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:25.169965+0000","last_change":"2024-03-30T13:56:45.121108+0000","last_active":"2024-03-30T13:57:25.169965+0000","last_peered":"2024-03-30T13:57:25.169965+0000","last_clean":"2024-03-30T13:57:25.169965+0000","last_became_active":"2024-03-30T13:56:45.120823+0000","last_became_peered":"2024-03-30T13:56:45.120823+0000","last_unstale":"2024-03-30T13:57:25.169965+0000","last_undegraded":"2024-03-30T13:57:25.169965+0000","last_fullsized":"2024-03-30T13:57:25.169965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T20:36:10.942984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204371+0000","last_change":"2024-03-30T13:56:47.126611+0000","last_active":"2024-03-30T13:57:24.204371+0000","last_peered":"2024-03-30T13:57:24.204371+0000","last_clean":"2024-03-30T13:57:24.204371+0000","last_became_active":"2024-03-30T13:56:47.126287+0000","last_became_peered":"2024-03-30T13:56:47.126287+0000","last_unstale":"2024-03-30T13:57:24.204371+0000","last_undegraded":"2024-03-30T13:57:24.204371+0000","last_fullsized":"2024-03-30T13:57:24.204371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:54:23.267905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.204619+0000","last_change":"2024-03-30T13:56:51.147528+0000","last_active":"2024-03-30T13:57:24.204619+0000","last_peered":"2024-03-30T13:57:24.204619+0000","last_clean":"2024-03-30T13:57:24.204619+0000","last_became_active":"2024-03-30T13:56:51.147147+0000","last_became_peered":"2024-03-30T13:56:51.147147+0000","last_unstale":"2024-03-30T13:57:24.204619+0000","last_undegraded":"2024-03-30T13:57:24.204619+0000","last_fullsized":"2024-03-30T13:57:24.204619+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:50.121039+0000","last_clean_scrub_stamp":"2024-03-30T13:56:50.121039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T21:23:07.720769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:58:05.175937+0000","last_change":"2024-03-30T13:56:49.136870+0000","last_active":"2024-03-30T13:58:05.175937+0000","last_peered":"2024-03-30T13:58:05.175937+0000","last_clean":"2024-03-30T13:58:05.175937+0000","last_became_active":"2024-03-30T13:56:49.136538+0000","last_became_peered":"2024-03-30T13:56:49.136538+0000","last_unstale":"2024-03-30T13:58:05.175937+0000","last_undegraded":"2024-03-30T13:58:05.175937+0000","last_fullsized":"2024-03-30T13:58:05.175937+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:48.109846+0000","last_clean_scrub_stamp":"2024-03-30T13:56:48.109846+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:58:15.275356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.612886+0000","last_change":"2024-03-30T13:56:45.119857+0000","last_active":"2024-03-30T13:57:24.612886+0000","last_peered":"2024-03-30T13:57:24.612886+0000","last_clean":"2024-03-30T13:57:24.612886+0000","last_became_active":"2024-03-30T13:56:45.119543+0000","last_became_peered":"2024-03-30T13:56:45.119543+0000","last_unstale":"2024-03-30T13:57:24.612886+0000","last_undegraded":"2024-03-30T13:57:24.612886+0000","last_fullsized":"2024-03-30T13:57:24.612886+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:44.102763+0000","last_clean_scrub_stamp":"2024-03-30T13:56:44.102763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:14:58.774300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T13:57:24.203580+0000","last_change":"2024-03-30T13:56:47.130841+0000","last_active":"2024-03-30T13:57:24.203580+0000","last_peered":"2024-03-30T13:57:24.203580+0000","last_clean":"2024-03-30T13:57:24.203580+0000","last_became_active":"2024-03-30T13:56:47.130484+0000","last_became_peered":"2024-03-30T13:56:47.130484+0000","last_unstale":"2024-03-30T13:57:24.203580+0000","last_undegraded":"2024-03-30T13:57:24.203580+0000","last_fullsized":"2024-03-30T13:57:24.203580+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T13:56:46.106043+0000","last_clean_scrub_stamp":"2024-03-30T13:56:46.106043+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:59:40.892832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":73,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1318912,"data_stored":873616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28280,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716872,"statfs":{"total":95995035648,"available":95966076928,"internally_reserved":0,"allocated":1290240,"data_stored":871291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":749568,"data_stored":282378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":873358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":704512,"data_stored":281271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":737280,"data_stored":281827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":737280,"data_stored":282814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":704512,"data_stored":281359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:58:12.869 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T13:58:12.869 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T13:58:12.869 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-30T13:58:12.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-30T13:58:13.111 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T13:58:14.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:14 smithi102 ceph-mon[29392]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:14.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:14 smithi102 ceph-mon[29392]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:14.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:14 smithi102 ceph-mon[37717]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:14.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:14 smithi102 ceph-mon[37717]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:14.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:14 smithi138 ceph-mon[33397]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T13:58:14.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:14 smithi138 ceph-mon[33397]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 -- 172.21.15.102:0/1547652864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c101140 msgr2=0x7fbc9c1015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 --2- 172.21.15.102:0/1547652864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c101140 0x7fbc9c1015c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0098f0 tx=0x7fbc8c009c00 comp rx=0 tx=0).stop 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 -- 172.21.15.102:0/1547652864 shutdown_connections 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 --2- 172.21.15.102:0/1547652864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 0x7fbc9c10dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 --2- 172.21.15.102:0/1547652864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbc9c1071f0 0x7fbc9c1075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 --2- 172.21.15.102:0/1547652864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c101140 0x7fbc9c1015c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 -- 172.21.15.102:0/1547652864 >> 172.21.15.102:0/1547652864 conn(0x7fbc9c078240 msgr2=0x7fbc9c078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 -- 172.21.15.102:0/1547652864 shutdown_connections 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 -- 172.21.15.102:0/1547652864 wait complete. 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 Processor -- start 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.791+0000 7fbca3698700 1 -- start start 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbc9c101140 0x7fbc9c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 0x7fbc9c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c1071f0 0x7fbc9c19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c110af0 con 0x7fbc9c1071f0 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c110970 con 0x7fbc9c101140 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9c110c70 con 0x7fbc9c101b00 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 0x7fbc9c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca1c35700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c1071f0 0x7fbc9c19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 0x7fbc9c1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:36214/0 (socket says 172.21.15.102:36214) 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca1c35700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c1071f0 0x7fbc9c19ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46926/0 (socket says 172.21.15.102:46926) 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 -- 172.21.15.102:0/1464519381 learned_addr learned my addr 172.21.15.102:0/1464519381 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T13:58:14.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca1434700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbc9c101140 0x7fbc9c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 -- 172.21.15.102:0/1464519381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbc9c101140 msgr2=0x7fbc9c198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbc9c101140 0x7fbc9c198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 -- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c1071f0 msgr2=0x7fbc9c19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c1071f0 0x7fbc9c19ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 -- 172.21.15.102:0/1464519381 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c19d170 con 0x7fbc9c101b00 2024-03-30T13:58:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca1c35700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c1071f0 0x7fbc9c19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca0c33700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 0x7fbc9c1986a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0059d0 tx=0x7fbc8c00fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbc927fc700 1 -- 172.21.15.102:0/1464519381 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c027070 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9c19d400 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbc927fc700 1 -- 172.21.15.102:0/1464519381 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fbc8c005570 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbc927fc700 1 -- 172.21.15.102:0/1464519381 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c004930 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.792+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc9c1a4ce0 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.793+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc9c068f00 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.794+0000 7fbc927fc700 1 -- 172.21.15.102:0/1464519381 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fbc8c02a070 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.794+0000 7fbc927fc700 1 --2- 172.21.15.102:0/1464519381 >> v2:172.21.15.102:6800/622906077 conn(0x7fbc8806dc00 0x7fbc880700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.794+0000 7fbca1434700 1 --2- 172.21.15.102:0/1464519381 >> v2:172.21.15.102:6800/622906077 conn(0x7fbc8806dc00 0x7fbc880700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.794+0000 7fbc927fc700 1 -- 172.21.15.102:0/1464519381 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 6259+0+0 (secure 0 0 0) 0x7fbc8c08e610 con 0x7fbc9c101b00 2024-03-30T13:58:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.795+0000 7fbca1434700 1 --2- 172.21.15.102:0/1464519381 >> v2:172.21.15.102:6800/622906077 conn(0x7fbc8806dc00 0x7fbc880700c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009be0 tx=0x7fbc98009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T13:58:14.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:14.799+0000 7fbc927fc700 1 -- 172.21.15.102:0/1464519381 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbc8c013030 con 0x7fbc9c101b00 2024-03-30T13:58:15.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.041+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbc9c1a4eb0 con 0x7fbc9c101b00 2024-03-30T13:58:15.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.041+0000 7fbc927fc700 1 -- 172.21.15.102:0/1464519381 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fbc8c05c0f0 con 0x7fbc9c101b00 2024-03-30T13:58:15.038 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-30T13:58:15.038 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 >> v2:172.21.15.102:6800/622906077 conn(0x7fbc8806dc00 msgr2=0x7fbc880700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 --2- 172.21.15.102:0/1464519381 >> v2:172.21.15.102:6800/622906077 conn(0x7fbc8806dc00 0x7fbc880700c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009be0 tx=0x7fbc98009260 comp rx=0 tx=0).stop 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 msgr2=0x7fbc9c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 0x7fbc9c1986a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0059d0 tx=0x7fbc8c00fa70 comp rx=0 tx=0).stop 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 shutdown_connections 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc9c1071f0 0x7fbc9c19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 --2- 172.21.15.102:0/1464519381 >> v2:172.21.15.102:6800/622906077 conn(0x7fbc8806dc00 0x7fbc880700c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbc9c101140 0x7fbc9c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 --2- 172.21.15.102:0/1464519381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc9c101b00 0x7fbc9c1986a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T13:58:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 >> 172.21.15.102:0/1464519381 conn(0x7fbc9c078240 msgr2=0x7fbc9c06a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T13:58:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 shutdown_connections 2024-03-30T13:58:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-30T13:58:15.043+0000 7fbca3698700 1 -- 172.21.15.102:0/1464519381 wait complete. 2024-03-30T13:58:15.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1464519381' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T13:58:15.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:58:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:58:15.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1464519381' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T13:58:15.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1464519381' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T13:58:15.771 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-30T13:58:15.771 INFO:tasks.cephadm:Setup complete, yielding 2024-03-30T13:58:15.771 INFO:teuthology.run_tasks:Running task workunit... 2024-03-30T13:58:15.787 INFO:tasks.workunit:Pulling workunits from ref 5af0400050a123dc31f105425174f106e057276a 2024-03-30T13:58:15.788 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-30T13:58:15.788 DEBUG:teuthology.orchestra.run.smithi102:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-30T13:58:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:58:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-30T13:58:15.804 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-30T13:58:15.861 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-30T13:58:15.861 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-30T13:58:15.917 INFO:tasks.workunit:timeout=3h 2024-03-30T13:58:15.917 INFO:tasks.workunit:cleanup=True 2024-03-30T13:58:15.917 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5af0400050a123dc31f105425174f106e057276a 2024-03-30T13:58:15.972 INFO:tasks.workunit.client.0.smithi102.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-30T13:58:16.339 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:16 smithi138 ceph-mon[33397]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:16.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:16 smithi102 ceph-mon[29392]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:16.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:16 smithi102 ceph-mon[37717]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:16.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:58:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:17.342 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:17.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:17.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:18 smithi138 ceph-mon[33397]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:18.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:18 smithi102 ceph-mon[29392]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:18.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:18 smithi102 ceph-mon[37717]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:19.184 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 systemd[1]: Starting Ceph grafana.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:58:19.188 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:18 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:18.900Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00255162s 2024-03-30T13:58:19.569 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 podman[64478]: 2024-03-30 13:58:19.158195613 +0000 UTC m=+0.019402059 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-30T13:58:19.569 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 podman[64478]: 2024-03-30 13:58:19.279890497 +0000 UTC m=+0.141096942 container create 2c0425b41c9a0ad35261b371a387b5a3230aa42caea299af9c3f1a58d2ec78ff (image=quay.io/ceph/grafana:9.4.12, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T13:58:19.569 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 podman[64478]: 2024-03-30 13:58:19.490648898 +0000 UTC m=+0.351855343 container init 2c0425b41c9a0ad35261b371a387b5a3230aa42caea299af9c3f1a58d2ec78ff (image=quay.io/ceph/grafana:9.4.12, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T13:58:19.569 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 podman[64478]: 2024-03-30 13:58:19.499532934 +0000 UTC m=+0.360739380 container start 2c0425b41c9a0ad35261b371a387b5a3230aa42caea299af9c3f1a58d2ec78ff (image=quay.io/ceph/grafana:9.4.12, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T13:58:19.820 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 bash[64478]: 2c0425b41c9a0ad35261b371a387b5a3230aa42caea299af9c3f1a58d2ec78ff 2024-03-30T13:58:19.820 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 systemd[1]: Started Ceph grafana.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:19.820 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652089957Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-30T13:58:19.820 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652376539Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-30T13:58:19.820 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652393686Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-30T13:58:19.820 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652399366Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.65240424Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652408954Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652413598Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652418106Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652422753Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652427825Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652433083Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652437565Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652442105Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652447164Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652451604Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652455895Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-30T13:58:19.821 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652460299Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-30T13:58:19.822 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652464998Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-30T13:58:19.822 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=settings t=2024-03-30T13:58:19.652470205Z level=info msg="App mode production" 2024-03-30T13:58:19.822 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=sqlstore t=2024-03-30T13:58:19.652532914Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=sqlstore t=2024-03-30T13:58:19.652549512Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.653845499Z level=info msg="Starting DB migrations" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.65393208Z level=info msg="Executing migration" id="create migration_log table" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.654708665Z level=info msg="Executing migration" id="create user table" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.655253209Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.656047004Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.65683071Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.657635504Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.658452987Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.659257047Z level=info msg="Executing migration" id="create user table v2" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.660022395Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.660833279Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-30T13:58:19.823 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.66167987Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.662454037Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.663242405Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.663758707Z level=info msg="Executing migration" id="Update user table charset" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.664548488Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.665108541Z level=info msg="Executing migration" id="Add missing user data" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.665489441Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.666264506Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.666682768Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.667090334Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.668467124Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.668933181Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.669550992Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.670305665Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.671121137Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-30T13:58:19.824 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.671917763Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.672764502Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.67357866Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.6744323Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.675203623Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.676012568Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.677099025Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.677879905Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.678701816Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.679518353Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-30T13:58:19.825 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.680293624Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.681123725Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.681890031Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.68443772Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.68485276Z level=info msg="Executing migration" id="create star table" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.687673446Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.690318854Z level=info msg="Executing migration" id="create org table v1" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.692939133Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.695545522Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.698224412Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-30T13:58:19.826 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.70090006Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.701470476Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.704164651Z level=info msg="Executing migration" id="Update org table charset" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.706769286Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.709398095Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.709850919Z level=info msg="Executing migration" id="create dashboard table" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.712506449Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.71508751Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.717684184Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.718225608Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.720757774Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.723315041Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.72610931Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.726653816Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.729314213Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-30T13:58:19.827 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.731977408Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.734420911Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.735134289Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.737573152Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.740316931Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.742765094Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.743518173Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.746159331Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.74863603Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.751321019Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.751847535Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.754461532Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.75717656Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.757965443Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-30T13:58:19.828 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.75873314Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.759527765Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.76032139Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.761149938Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.761746749Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.762551448Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.763371003Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.764173417Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.764744704Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.766494441Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.767225058Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.768015349Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.768572692Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.769354642Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.769875842Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-30T13:58:19.829 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.770633475Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-30T13:58:19.830 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.77143537Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-30T13:58:19.830 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.771854401Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-30T13:58:19.830 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.774499621Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-30T13:58:19.830 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.777227285Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-30T13:58:19.830 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.779940309Z level=info msg="Executing migration" id="create data_source table" 2024-03-30T13:58:19.830 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.780583515Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-30T13:58:19.830 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.783136248Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-30T13:58:19.831 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.783696286Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-30T13:58:19.831 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.78648399Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-30T13:58:19.831 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.787024098Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-30T13:58:19.831 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.789667323Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-30T13:58:19.831 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.7922771Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.794999567Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.79559988Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.798025264Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.7988452Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.800609069Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.803140973Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.803574685Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-30T13:58:19.832 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.80608043Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.808563275Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.811246898Z level=info msg="Executing migration" id="Add uid column" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.812056657Z level=info msg="Executing migration" id="Update uid value" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.814656618Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.817161578Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.81957564Z level=info msg="Executing migration" id="create api_key table" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.820134224Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.820925695Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.821740223Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.822551787Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.823076904Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.823878908Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.824705725Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-30T13:58:19.833 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.826576776Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-30T13:58:19.834 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.827331125Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-30T13:58:19.834 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.827874593Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-30T13:58:19.834 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.828434449Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-30T13:58:19.834 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.828999506Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-30T13:58:20.072 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-mon[33397]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:20.072 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.072 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.072 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.072 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.073 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.82942881Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-30T13:58:20.073 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.829881469Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-30T13:58:20.073 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.830241267Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.831019577Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.83181584Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.832209286Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.832981856Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.833790708Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.834297925Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.834766623Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.835292471Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-30T13:58:20.074 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.835824221Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.836362132Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.836891346Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.837273874Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.837636781Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.838461116Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.83929287Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.839677414Z level=info msg="Executing migration" id="create quota table v1" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.840183286Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.840738452Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.841083345Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.841607421Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-30T13:58:20.075 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.842133363Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.842966813Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.84335619Z level=info msg="Executing migration" id="create session table" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.843971989Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.844358285Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.844741737Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.84527799Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.845759267Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.846103196Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.846466277Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.846811559Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.847172111Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.847692123Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-30T13:58:20.076 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.848023617Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.848911265Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.849333237Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.850239852Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.851127483Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.85149756Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.852039034Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.852627649Z level=info msg="Executing migration" id="create alert table v1" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.853447373Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.854186363Z level=info msg="Executing migration" id="add index alert state" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.854945731Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.85548427Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-30T13:58:20.077 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.855967477Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.856526749Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.857082131Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.85971794Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.86025908Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.860808265Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.861221106Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.86170433Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.862243199Z level=info msg="Executing migration" id="Add column is_default" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.863246664Z level=info msg="Executing migration" id="Add column frequency" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.864255777Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.865238302Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-30T13:58:20.078 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.866221158Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.866756347Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.867129511Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.867498424Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.867975471Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.868525628Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.869073197Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.869600889Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.870132706Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.871173788Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-30T13:58:20.079 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.872207182Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-30T13:58:20.080 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.87298286Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-30T13:58:20.080 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.873837156Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-30T13:58:20.080 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.874715748Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-30T13:58:20.080 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.875508027Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-30T13:58:20.080 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.876307001Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-30T13:58:20.081 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.877227621Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-30T13:58:20.081 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.878111629Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-30T13:58:20.081 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.878931452Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-30T13:58:20.081 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.87970009Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-30T13:58:20.081 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.88051602Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.881326548Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.882149012Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.882999375Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.883791733Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.88463712Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.885416562Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.885843752Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.886668772Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.887537279Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.890244971Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.892876901Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.895522046Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-30T13:58:20.082 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.898330118Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.90113873Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.903733649Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.906436133Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.909084361Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.911827801Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.914570178Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.916975368Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.919461832Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.919778484Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.920587487Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.921403652Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.922294909Z level=info msg="Executing migration" id="Move region to single row" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.923108118Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-30T13:58:20.083 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.92389281Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.924684223Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.925538259Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.92632248Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.927171804Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.927958785Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.928779491Z level=info msg="Executing migration" id="create test_data table" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.929589777Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.930354809Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.931196682Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.932041814Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.932860778Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.933653841Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.936181383Z level=info msg="Executing migration" id="create team table" 2024-03-30T13:58:20.084 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.938790352Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.941655913Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.944211298Z level=info msg="Executing migration" id="create team member table" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.946847155Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.949484383Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.95215214Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.954817223Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.957436273Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.960062998Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.962702155Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.965187463Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.967802453Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.970495432Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.97299244Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.975760623Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-30T13:58:20.085 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.978324703Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.978773984Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.981283962Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.983848382Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.986224354Z level=info msg="Executing migration" id="create tag table" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.986613581Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.989219211Z level=info msg="Executing migration" id="create login attempt table" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.991883244Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.994525344Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.994956207Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:19 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:19.997440042Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.000152253Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.002772224Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.003127323Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.005685847Z level=info msg="Executing migration" id="create user auth table" 2024-03-30T13:58:20.086 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.008328703Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.010931943Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.011235765Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.01362906Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.014500732Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.015363893Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.016235658Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.017017385Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.017896517Z level=info msg="Executing migration" id="create server_lock table" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.018673514Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.019105606Z level=info msg="Executing migration" id="create user auth token table" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.019877619Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.020718272Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.021497626Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-30T13:58:20.087 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.021926239Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.02280406Z level=info msg="Executing migration" id="create cache_data table" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.023591228Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.02443078Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.02489247Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.025728764Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.026479845Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.027318789Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.027808429Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.028591971Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.02945765Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.030261767Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.030719781Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.031498668Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.03193715Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-30T13:58:20.088 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.032368786Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.03324569Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.03366947Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.034428643Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.034839277Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.035269759Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.036086786Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.036380654Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.037190579Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.037623873Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.038017364Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-30T13:58:20.089 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.03868892Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.040745112Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.04116735Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.043773967Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.048168126Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.052621589Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.053082981Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.055867303Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.058390413Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.060973263Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.061472251Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-30T13:58:20.090 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.064001282Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-30T13:58:20.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.066778464Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-30T13:58:20.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.069302862Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-30T13:58:20.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.072061074Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-30T13:58:20.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.074644841Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-30T13:58:20.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.077114285Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-30T13:58:20.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.079628347Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-30T13:58:20.091 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.082063297Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-30T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[37717]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[29392]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:20.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:20 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.084865797Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.087483805Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.090111863Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.092735268Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.095369993Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.098019872Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.100653812Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.103381904Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.105796542Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.10849759Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-30T13:58:20.324 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.111051729Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-30T13:58:20.325 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.113722223Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-30T13:58:20.325 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.11622091Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-30T13:58:20.325 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.118771064Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-30T13:58:20.325 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.12131703Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-30T13:58:20.325 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.122294613Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-30T13:58:20.325 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.125047702Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.127841228Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.130541223Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.133148435Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.135676959Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.138268708Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.140944171Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.143687818Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.146195825Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-30T13:58:20.326 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.148787866Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.151468011Z level=info msg="Executing migration" id="create alert_image table" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.154235229Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.156874803Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.159531492Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.159937441Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.162104613Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.162176882Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.164811511Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.167458148Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.169940746Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.170070337Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.170173196Z level=info msg="alerts found to migrate" alerts=0 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.172505395Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-30T13:58:20.327 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.175278602Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.177880197Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.180421524Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.182937443Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.185405052Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.187910613Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.190386247Z level=info msg="Executing migration" id="create data_keys table" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.192868155Z level=info msg="Executing migration" id="create secrets table" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.195261879Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.200785255Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.203475964Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.205917981Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.211831951Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.218144494Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-30T13:58:20.328 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.218924219Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.219706705Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.220530327Z level=info msg="Executing migration" id="create permission table" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.221349416Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.222167022Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.222958594Z level=info msg="Executing migration" id="create role table" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.223777859Z level=info msg="Executing migration" id="add column display_name" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.224847488Z level=info msg="Executing migration" id="add column group_name" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.225917674Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.227422108Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.228210159Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.228998916Z level=info msg="Executing migration" id="create team role table" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.229814341Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-30T13:58:20.329 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.230619788Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.231392815Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.232187659Z level=info msg="Executing migration" id="create user role table" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.232966303Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.233792366Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.23467828Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.235517825Z level=info msg="Executing migration" id="create builtin role table" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.236282207Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.237168509Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.238021104Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.239143736Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.239663519Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.240417352Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-30T13:58:20.330 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.240836029Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.241265047Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.24164904Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.242074183Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.243177548Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.243964211Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.244409063Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.244687685Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.245088863Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.245523011Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.245849234Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.246118047Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.246463103Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.246837534Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-30T13:58:20.331 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.247286002Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.248410026Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.248698818Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.249102559Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.249521482Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.249928139Z level=info msg="Executing migration" id="add correlation config column" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.251036881Z level=info msg="Executing migration" id="create entity_events table" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.251446758Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.252222468Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.252302527Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.252566521Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.252640125Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.252876595Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.253298734Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-30T13:58:20.332 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.253717278Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-30T13:58:20.333 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.25454953Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T13:58:20.333 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.25543767Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-30T13:58:20.333 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.256287871Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-30T13:58:20.333 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.257106797Z level=info msg="Executing migration" id="Drop public config table" 2024-03-30T13:58:20.333 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.257930398Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-30T13:58:20.333 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.258761296Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-30T13:58:20.333 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.26138323Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.264152971Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.266776132Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.270560092Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.273180641Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.275777367Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.278416137Z level=info msg="Executing migration" id="add share column" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.281050801Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.283720701Z level=info msg="Executing migration" id="create file table" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.286318758Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.288763993Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.291216157Z level=info msg="Executing migration" id="create file_meta table" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.291616088Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-30T13:58:20.334 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.294244775Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.29700647Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.299641128Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.302175039Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.304759134Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.30754725Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.310194475Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.312831614Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.315362342Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.318146886Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.320821317Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.323418469Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.325938502Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-30T13:58:20.335 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.328393124Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-30T13:58:20.633 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:20 smithi102 systemd[1]: Starting Ceph node-exporter.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.342942238Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.345428329Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.347924403Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.352249932Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.355167165Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.357736416Z level=info msg="Executing migration" id="create folder table" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.360265163Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.362693126Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.365341828Z level=info msg="Executing migration" id="Update folder title length" 2024-03-30T13:58:20.685 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.366177543Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-30T13:58:20.686 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=migrator t=2024-03-30T13:58:20.368441637Z level=info msg="migrations completed" performed=485 skipped=0 duration=714.533734ms 2024-03-30T13:58:20.686 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=sqlstore t=2024-03-30T13:58:20.369016672Z level=info msg="Created default organization" 2024-03-30T13:58:20.686 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=plugin.finder t=2024-03-30T13:58:20.402527366Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-30T13:58:20.686 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=secrets t=2024-03-30T13:58:20.402613285Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-30T13:58:20.686 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=query_data t=2024-03-30T13:58:20.404687817Z level=info msg="Query Service initialization" 2024-03-30T13:58:20.686 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=live.push_http t=2024-03-30T13:58:20.407531806Z level=info msg="Live Push Gateway initialization" 2024-03-30T13:58:20.938 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:20 smithi102 bash[73069]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-30T13:58:21.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-mon[33397]: Deploying daemon node-exporter.a on smithi102 2024-03-30T13:58:21.435 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=infra.usagestats.collector t=2024-03-30T13:58:21.00611502Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-30T13:58:21.435 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=provisioning.datasources t=2024-03-30T13:58:21.007134479Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-30T13:58:21.435 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=provisioning.datasources t=2024-03-30T13:58:21.018204415Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-30T13:58:21.435 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=provisioning.alerting t=2024-03-30T13:58:21.028747642Z level=info msg="starting to provision alerting" 2024-03-30T13:58:21.435 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=provisioning.alerting t=2024-03-30T13:58:21.028757281Z level=info msg="finished to provision alerting" 2024-03-30T13:58:21.435 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=grafanaStorageLogger t=2024-03-30T13:58:21.02900036Z level=info msg="storage starting" 2024-03-30T13:58:21.436 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=ngalert.state.manager t=2024-03-30T13:58:21.029199356Z level=info msg="Warming state cache for startup" 2024-03-30T13:58:21.436 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=ngalert.state.manager t=2024-03-30T13:58:21.029416288Z level=info msg="State cache has been initialized" states=0 duration=216.027µs 2024-03-30T13:58:21.436 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=ticker t=2024-03-30T13:58:21.029467254Z level=info msg=starting first_tick=2024-03-30T13:58:30Z 2024-03-30T13:58:21.436 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=ngalert.multiorg.alertmanager t=2024-03-30T13:58:21.029481197Z level=info msg="starting MultiOrg Alertmanager" 2024-03-30T13:58:21.436 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=http.server t=2024-03-30T13:58:21.030854242Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-30T13:58:21.436 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 13:58:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=sqlstore.transactions t=2024-03-30T13:58:21.04996183Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T13:58:21.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:21 smithi102 ceph-mon[29392]: Deploying daemon node-exporter.a on smithi102 2024-03-30T13:58:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:21 smithi102 ceph-mon[37717]: Deploying daemon node-exporter.a on smithi102 2024-03-30T13:58:21.438 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:21 smithi102 bash[73069]: Getting image source signatures 2024-03-30T13:58:21.438 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:21 smithi102 bash[73069]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-30T13:58:21.438 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:21 smithi102 bash[73069]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-30T13:58:21.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:21 smithi102 bash[73069]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-30T13:58:22.339 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:22 smithi102 ceph-mon[29392]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:22.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:22 smithi102 ceph-mon[37717]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:22.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:22 smithi138 ceph-mon[33397]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:22.938 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:22 smithi102 bash[73069]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-30T13:58:22.938 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:22 smithi102 bash[73069]: Writing manifest to image destination 2024-03-30T13:58:23.438 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 podman[73069]: 2024-03-30 13:58:23.080739406 +0000 UTC m=+2.344873401 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 podman[73069]: 2024-03-30 13:58:23.179665926 +0000 UTC m=+2.443799906 container create 3a12e95ee124c409ff60de3f4a33196746318f51be1cff173671b17e00c35258 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 podman[73069]: 2024-03-30 13:58:23.398502127 +0000 UTC m=+2.662636120 container init 3a12e95ee124c409ff60de3f4a33196746318f51be1cff173671b17e00c35258 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 podman[73069]: 2024-03-30 13:58:23.40112083 +0000 UTC m=+2.665254813 container start 3a12e95ee124c409ff60de3f4a33196746318f51be1cff173671b17e00c35258 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.405Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.405Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.405Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.405Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.405Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.405Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=arp 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-30T13:58:23.439 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-30T13:58:23.440 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-30T13:58:23.440 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-30T13:58:23.440 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-30T13:58:23.440 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-30T13:58:23.440 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=edac 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-30T13:58:23.441 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=os 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-30T13:58:23.442 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=stat 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=time 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=uname 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-30T13:58:23.443 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a[73130]: ts=2024-03-30T13:58:23.406Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-30T13:58:23.938 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 bash[73069]: 3a12e95ee124c409ff60de3f4a33196746318f51be1cff173671b17e00c35258 2024-03-30T13:58:23.938 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 13:58:23 smithi102 systemd[1]: Started Ceph node-exporter.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:24.362 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:24 smithi138 ceph-mon[33397]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:24.362 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.362 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.362 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[37717]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[29392]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:24.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:24.684 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:24 smithi138 systemd[1]: Starting Ceph node-exporter.b for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:58:25.078 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:24 smithi138 bash[64766]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-30T13:58:25.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:58:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:58:25.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:25 smithi138 ceph-mon[33397]: Deploying daemon node-exporter.b on smithi138 2024-03-30T13:58:25.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:25 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:25.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:25 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:25.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:25 smithi138 bash[64766]: Getting image source signatures 2024-03-30T13:58:25.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:25 smithi138 bash[64766]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-30T13:58:25.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:25 smithi138 bash[64766]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-30T13:58:25.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:25 smithi138 bash[64766]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-30T13:58:25.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:25 smithi102 ceph-mon[29392]: Deploying daemon node-exporter.b on smithi138 2024-03-30T13:58:25.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:25 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:25.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:25 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:25.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:25 smithi102 ceph-mon[37717]: Deploying daemon node-exporter.b on smithi138 2024-03-30T13:58:25.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:25 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:25.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:25 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:26.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:26 smithi138 ceph-mon[33397]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:26.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:26 smithi102 ceph-mon[29392]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:26.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:26 smithi102 ceph-mon[37717]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:26.922 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:26 smithi138 bash[64766]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-30T13:58:26.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:58:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:27.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:27.184 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:26 smithi138 bash[64766]: Writing manifest to image destination 2024-03-30T13:58:27.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:28.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:28 smithi138 ceph-mon[33397]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:28.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:28 smithi102 ceph-mon[29392]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:28.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:28 smithi102 ceph-mon[37717]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:29.911 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:29 smithi138 podman[64766]: 2024-03-30 13:58:29.527786542 +0000 UTC m=+4.875301309 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-30T13:58:29.911 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:29 smithi138 podman[64766]: 2024-03-30 13:58:29.79281318 +0000 UTC m=+5.140327984 container create 560ebe94c64ed21d590fb4e4de34f76be86b20574c695331b28679bb138fb50d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T13:58:30.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-mon[33397]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 podman[64766]: 2024-03-30 13:58:30.188806934 +0000 UTC m=+5.536321697 container init 560ebe94c64ed21d590fb4e4de34f76be86b20574c695331b28679bb138fb50d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 podman[64766]: 2024-03-30 13:58:30.191824828 +0000 UTC m=+5.539339593 container start 560ebe94c64ed21d590fb4e4de34f76be86b20574c695331b28679bb138fb50d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.196Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.196Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=arp 2024-03-30T13:58:30.435 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=edac 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-30T13:58:30.436 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=os 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-30T13:58:30.437 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-30T13:58:30.438 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-30T13:58:30.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:30 smithi102 ceph-mon[37717]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:30.438 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-30T13:58:30.438 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=stat 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=time 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=uname 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-30T13:58:30.439 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.197Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-30T13:58:30.440 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.198Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-30T13:58:30.440 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b[64816]: ts=2024-03-30T13:58:30.198Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-30T13:58:30.440 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 bash[64766]: 560ebe94c64ed21d590fb4e4de34f76be86b20574c695331b28679bb138fb50d 2024-03-30T13:58:30.440 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 13:58:30 smithi138 systemd[1]: Started Ceph node-exporter.b for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:30.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:30 smithi102 ceph-mon[29392]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:31.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:31 smithi138 ceph-mon[33397]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:31.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.667 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.667 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.667 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.667 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:31 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:58:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[29392]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[37717]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:31 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:58:33.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:33 smithi102 ceph-mon[29392]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:33.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:33 smithi102 ceph-mon[37717]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:33.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:33 smithi138 ceph-mon[33397]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:34.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:34 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:34 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:34 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:34 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:34 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:34 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:34 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:34 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:34 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:35.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:58:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:58:36.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[29392]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:36.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:36.057 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:36.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[37717]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:36.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:36.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:36.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:35 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:35 smithi138 ceph-mon[33397]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:35 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:35 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:35 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:35 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:36.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:35 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:36.686 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:58:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:36.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:36.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:36 smithi102 ceph-mon[29392]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T13:58:36.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:36 smithi102 ceph-mon[29392]: Reconfiguring daemon alertmanager.a on smithi102 2024-03-30T13:58:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:36 smithi102 ceph-mon[37717]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T13:58:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:36 smithi102 ceph-mon[37717]: Reconfiguring daemon alertmanager.a on smithi102 2024-03-30T13:58:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:36 smithi138 ceph-mon[33397]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T13:58:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:36 smithi138 ceph-mon[33397]: Reconfiguring daemon alertmanager.a on smithi102 2024-03-30T13:58:37.793 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:37 smithi102 ceph-mon[37717]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:37.793 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:37 smithi102 ceph-mon[29392]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:38.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:37 smithi138 ceph-mon[33397]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:39.188 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:39 smithi102 systemd[1]: Stopping Ceph alertmanager.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:58:39.630 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:39 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[72365]: ts=2024-03-30T13:58:39.325Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-30T13:58:39.630 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:39 smithi102 podman[73846]: 2024-03-30 13:58:39.345839555 +0000 UTC m=+0.121254627 container died ff4acdb16fe1ec53ba463b4fcf0f7fa6ded500e6629e0232425a66587aac6b90 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:39.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:39 smithi102 ceph-mon[29392]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:39.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:39 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:39 smithi102 ceph-mon[37717]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:39 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:39.938 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:39 smithi102 podman[73846]: 2024-03-30 13:58:39.733535799 +0000 UTC m=+0.508950860 container cleanup ff4acdb16fe1ec53ba463b4fcf0f7fa6ded500e6629e0232425a66587aac6b90 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:39.938 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:39 smithi102 bash[73846]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a 2024-03-30T13:58:40.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:39 smithi138 ceph-mon[33397]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:40.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:39 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:40.286 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 podman[73860]: 2024-03-30 13:58:40.035354768 +0000 UTC m=+0.689836467 container remove ff4acdb16fe1ec53ba463b4fcf0f7fa6ded500e6629e0232425a66587aac6b90 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:40.286 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 podman[73860]: 2024-03-30 13:58:40.127751226 +0000 UTC m=+0.782232925 volume remove ef960bd5f030ab1a091e1bff3359aa7db93ea4c4065f6982489bbc40f210d672 2024-03-30T13:58:40.641 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-03-30T13:58:40.641 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 systemd[1]: Stopped Ceph alertmanager.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:40.641 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 systemd[1]: Starting Ceph alertmanager.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:58:40.641 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 podman[73956]: 2024-03-30 13:58:40.426550282 +0000 UTC m=+0.018124847 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-30T13:58:40.641 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 podman[73956]: 2024-03-30 13:58:40.545685187 +0000 UTC m=+0.137259786 volume create 4f750ed12b02f09942bc7ab9119f688621952f12c6f5fb74cfac9c0137b21da2 2024-03-30T13:58:40.642 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 podman[73956]: 2024-03-30 13:58:40.64622238 +0000 UTC m=+0.237796936 container create 4ebdf376ccc74ca162597e215d36114ee413ded109c933cb7a859fb4a60e8010 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:41.438 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 podman[73956]: 2024-03-30 13:58:40.991524177 +0000 UTC m=+0.583098746 container init 4ebdf376ccc74ca162597e215d36114ee413ded109c933cb7a859fb4a60e8010 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:41.438 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:40 smithi102 podman[73956]: 2024-03-30 13:58:40.994166285 +0000 UTC m=+0.585740853 container start 4ebdf376ccc74ca162597e215d36114ee413ded109c933cb7a859fb4a60e8010 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.006Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.006Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.007Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.008Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.036Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.036Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.039Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:41.039Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 bash[73956]: 4ebdf376ccc74ca162597e215d36114ee413ded109c933cb7a859fb4a60e8010 2024-03-30T13:58:41.439 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:41 smithi102 systemd[1]: Started Ceph alertmanager.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:42.065 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:41 smithi138 ceph-mon[33397]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:42.065 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:42.065 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:42.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-mon[29392]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:42.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:42.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:42.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-mon[37717]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:42.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:42.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:43.148 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:42 smithi138 ceph-mon[33397]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T13:58:43.148 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:42 smithi138 ceph-mon[33397]: Reconfiguring daemon prometheus.a on smithi138 2024-03-30T13:58:43.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:42 smithi102 ceph-mon[29392]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T13:58:43.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:42 smithi102 ceph-mon[29392]: Reconfiguring daemon prometheus.a on smithi138 2024-03-30T13:58:43.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:42 smithi102 ceph-mon[37717]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T13:58:43.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:42 smithi102 ceph-mon[37717]: Reconfiguring daemon prometheus.a on smithi138 2024-03-30T13:58:43.188 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:43 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:43.009Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000721521s 2024-03-30T13:58:44.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:43 smithi138 ceph-mon[33397]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:43 smithi102 ceph-mon[29392]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:44.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:43 smithi102 ceph-mon[37717]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:44.846 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 systemd[1]: Stopping Ceph prometheus.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:58:45.138 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-30T13:58:45.138 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-30T13:58:45.138 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.851Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.852Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.852Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[62513]: ts=2024-03-30T13:58:44.852Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:44 smithi138 podman[65724]: 2024-03-30 13:58:44.861557604 +0000 UTC m=+0.123594118 container died ce20971fd7ba01703e62de2658a28a88cf58d76b017b589059168e00fd8ebac1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 podman[65724]: 2024-03-30 13:58:45.142989317 +0000 UTC m=+0.405025822 container cleanup ce20971fd7ba01703e62de2658a28a88cf58d76b017b589059168e00fd8ebac1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:45.139 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 bash[65724]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a 2024-03-30T13:58:45.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:58:45.434 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 podman[65739]: 2024-03-30 13:58:45.361593253 +0000 UTC m=+0.502297665 container remove ce20971fd7ba01703e62de2658a28a88cf58d76b017b589059168e00fd8ebac1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:45.754 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-03-30T13:58:45.754 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 systemd[1]: Stopped Ceph prometheus.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:45.754 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@prometheus.a.service: Consumed 1.064s CPU time. 2024-03-30T13:58:45.754 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 systemd[1]: Starting Ceph prometheus.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T13:58:45.754 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 podman[65834]: 2024-03-30 13:58:45.57632586 +0000 UTC m=+0.018916920 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-30T13:58:45.754 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 podman[65834]: 2024-03-30 13:58:45.679951462 +0000 UTC m=+0.122542563 container create 3ddcfe68deb6a8517311a39d97a4ae4829adac4ec061af119d9723eaa489fb8e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:46.034 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:45 smithi102 ceph-mon[37717]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:46.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:45 smithi102 ceph-mon[29392]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:46.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-mon[33397]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:46.184 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 podman[65834]: 2024-03-30 13:58:45.874118897 +0000 UTC m=+0.316709959 container init 3ddcfe68deb6a8517311a39d97a4ae4829adac4ec061af119d9723eaa489fb8e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:46.184 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 podman[65834]: 2024-03-30 13:58:45.877013846 +0000 UTC m=+0.319604905 container start 3ddcfe68deb6a8517311a39d97a4ae4829adac4ec061af119d9723eaa489fb8e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.900Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.900Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.900Z caller=main.go:570 level=info host_details="(Linux 5.14.0-434.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024 x86_64 smithi138 (none))" 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.900Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.900Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.901Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.902Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.904Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.904Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.905Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.905Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.521µs 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.905Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.905Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-30T13:58:46.185 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.905Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.905Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.238µs wal_replay_duration=446.991µs wbl_replay_duration=96ns total_replay_duration=483.901µs 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.906Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.906Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.906Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.923Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.222778ms db_storage=711ns remote_storage=1.111µs web_handler=306ns query_engine=675ns scrape=383.717µs scrape_sd=65.356µs notify=10.197µs notify_sd=14.135µs rules=16.352237ms tracing=5.448µs 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.923Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T13:58:45.923Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 bash[65834]: 3ddcfe68deb6a8517311a39d97a4ae4829adac4ec061af119d9723eaa489fb8e 2024-03-30T13:58:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 13:58:45 smithi138 systemd[1]: Started Ceph prometheus.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T13:58:46.381 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STOPPING 2024-03-30T13:58:46.381 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STOPPED 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STARTING 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Serving on http://:::9283 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STARTED 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STOPPING 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STOPPED 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STARTING 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Serving on http://:::9283 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STARTED 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STOPPING 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T13:58:46.382 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STOPPED 2024-03-30T13:58:46.383 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STARTING 2024-03-30T13:58:46.383 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Serving on http://:::9283 2024-03-30T13:58:46.383 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:13:58:46] ENGINE Bus STARTED 2024-03-30T13:58:47.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:47.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:58:47.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:58:47.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:46 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:58:47.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:47.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:58:47.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:58:47.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:58:47.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi102.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T13:58:47.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:58:47.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T13:58:47.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:58:47.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi138.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T13:58:47.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:47.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:46 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:58:48.413 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:48 smithi138 ceph-mon[33397]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:48 smithi102 ceph-mon[29392]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:48 smithi102 ceph-mon[37717]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:50.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:50 smithi138 ceph-mon[33397]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:50.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:50 smithi102 ceph-mon[29392]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:50.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:50 smithi102 ceph-mon[37717]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:58:51.300 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T13:58:51.013Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004012376s 2024-03-30T13:58:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:51 smithi138 ceph-mon[33397]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[37717]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:51.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[29392]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:51.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:52 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:52 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:52 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:52 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:52.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:52 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:52.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:52 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:52 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T13:58:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:52 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T13:58:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:52 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:58:53.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:53 smithi138 ceph-mon[33397]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:53.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:53 smithi102 ceph-mon[37717]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:53.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:53 smithi102 ceph-mon[29392]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:54.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:54 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:54.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:54 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:54.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:54 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:58:55.642 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:58:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:58:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:55 smithi138 ceph-mon[33397]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:55 smithi102 ceph-mon[37717]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:55.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:55 smithi102 ceph-mon[29392]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:56.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:56.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:58:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:58:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:58:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:58:57.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:57 smithi138 ceph-mon[33397]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:57.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:57 smithi102 ceph-mon[29392]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:57.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:57 smithi102 ceph-mon[37717]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:58:59.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:58:59 smithi138 ceph-mon[33397]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:58:59 smithi102 ceph-mon[37717]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:58:59.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:58:59 smithi102 ceph-mon[29392]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T13:59:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:01 smithi138 ceph-mon[33397]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:01.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:01 smithi102 ceph-mon[37717]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:01.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:01 smithi102 ceph-mon[29392]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:03.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:03 smithi138 ceph-mon[33397]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:03.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:03 smithi102 ceph-mon[37717]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:03.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:03 smithi102 ceph-mon[29392]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:05.657 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:59:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:59:05.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:05 smithi138 ceph-mon[33397]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:05 smithi102 ceph-mon[29392]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:05.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:05 smithi102 ceph-mon[37717]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:06.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:06.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:06.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:59:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:59:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:07.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:07 smithi138 ceph-mon[33397]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:07 smithi102 ceph-mon[29392]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:07.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:07 smithi102 ceph-mon[37717]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:09.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:09 smithi138 ceph-mon[33397]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:09.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:09 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:09.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:09 smithi102 ceph-mon[37717]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:09.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:09 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:09.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:09 smithi102 ceph-mon[29392]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:09.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:09 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:11.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:11 smithi138 ceph-mon[33397]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:11 smithi102 ceph-mon[37717]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:11.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:11 smithi102 ceph-mon[29392]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:13.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:13 smithi138 ceph-mon[33397]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:13.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:13 smithi102 ceph-mon[37717]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:13 smithi102 ceph-mon[29392]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:15.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:59:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:59:16.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:15 smithi138 ceph-mon[33397]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:15 smithi102 ceph-mon[29392]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:16.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:15 smithi102 ceph-mon[37717]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:16.921 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:59:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:59:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:17.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:17 smithi138 ceph-mon[33397]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:17 smithi102 ceph-mon[29392]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:18.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:17 smithi102 ceph-mon[37717]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:20.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:19 smithi138 ceph-mon[33397]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:19 smithi102 ceph-mon[29392]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:19 smithi102 ceph-mon[37717]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:22.105 INFO:tasks.workunit.client.0.smithi102.stderr:Note: switching to '5af0400050a123dc31f105425174f106e057276a'. 2024-03-30T13:59:22.105 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr: git switch -c 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr:Or undo this operation with: 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr: git switch - 2024-03-30T13:59:22.106 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-30T13:59:22.107 INFO:tasks.workunit.client.0.smithi102.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T13:59:22.107 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-03-30T13:59:22.107 INFO:tasks.workunit.client.0.smithi102.stderr:HEAD is now at 5af0400050a Merge pull request #56547 from rhcs-dashboard/wip-65193-squid 2024-03-30T13:59:22.110 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-30T13:59:22.142 INFO:tasks.workunit.client.0.smithi102.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-30T13:59:22.142 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T13:59:22.142 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-30T13:59:22.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:21 smithi138 ceph-mon[33397]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:21 smithi102 ceph-mon[29392]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:21 smithi102 ceph-mon[37717]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:22.927 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-30T13:59:22.999 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-30T13:59:23.038 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T13:59:23.040 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T13:59:23.040 INFO:tasks.workunit.client.0.smithi102.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-30T13:59:23.079 INFO:tasks.workunit.client.0.smithi102.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T13:59:23.082 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-30T13:59:23.082 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-30T13:59:23.138 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-30T13:59:23.139 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-30T13:59:23.139 DEBUG:teuthology.orchestra.run.smithi102:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5af0400050a123dc31f105425174f106e057276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-30T13:59:23.203 INFO:tasks.workunit.client.0.smithi102.stderr:+ parallel=1 2024-03-30T13:59:23.203 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' '' = --serial ']' 2024-03-30T13:59:23.203 INFO:tasks.workunit.client.0.smithi102.stderr:+ crimson=0 2024-03-30T13:59:23.203 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' '' = --crimson ']' 2024-03-30T13:59:23.203 INFO:tasks.workunit.client.0.smithi102.stderr:+ color= 2024-03-30T13:59:23.203 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -t 1 ']' 2024-03-30T13:59:23.203 INFO:tasks.workunit.client.0.smithi102.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-30T13:59:23.204 INFO:tasks.workunit.client.0.smithi102.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-30T13:59:23.204 INFO:tasks.workunit.client.0.smithi102.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-30T13:59:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:+ declare -A pids 2024-03-30T13:59:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_aio 2024-03-30T13:59:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_aio' 2024-03-30T13:59:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_aio 2024-03-30T13:59:23.205 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.206 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_aio 2024-03-30T13:59:23.206 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74736 2024-03-30T13:59:23.206 INFO:tasks.workunit.client.0.smithi102.stdout:test api_aio on pid 74736 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_aio on pid 74736' 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74736 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_aio_pp 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_aio_pp' 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_aio_pp 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.210 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.211 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_aio_pp 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74743 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_aio_pp on pid 74743' 2024-03-30T13:59:23.212 INFO:tasks.workunit.client.0.smithi102.stdout:test api_aio_pp on pid 74743 2024-03-30T13:59:23.216 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74743 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_io 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_io' 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_io 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.217 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_io 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74752 2024-03-30T13:59:23.220 INFO:tasks.workunit.client.0.smithi102.stdout:test api_io on pid 74752 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_io on pid 74752' 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74752 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_io_pp 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_io_pp' 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_io_pp 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.227 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.228 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.229 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_io_pp 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74763 2024-03-30T13:59:23.230 INFO:tasks.workunit.client.0.smithi102.stdout:test api_io_pp on pid 74763 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_io_pp on pid 74763' 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74763 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_asio 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_asio' 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_asio 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.236 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_asio 2024-03-30T13:59:23.242 INFO:tasks.workunit.client.0.smithi102.stdout:test api_asio on pid 74777 2024-03-30T13:59:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74777 2024-03-30T13:59:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_asio on pid 74777' 2024-03-30T13:59:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74777 2024-03-30T13:59:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_list 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_list' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_list 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.250 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.251 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.252 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.253 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.254 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_list 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74736/exe 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74795 2024-03-30T13:59:23.256 INFO:tasks.workunit.client.0.smithi102.stdout:test api_list on pid 74795 2024-03-30T13:59:23.261 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_list on pid 74795' 2024-03-30T13:59:23.261 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74795 2024-03-30T13:59:23.261 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_lock 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_lock' 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_lock 2024-03-30T13:59:23.262 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.264 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.265 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.266 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.268 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_aio: /' 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-30T13:59:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74743/exe 2024-03-30T13:59:23.270 INFO:tasks.workunit.client.0.smithi102.stdout:test api_lock on pid 74815 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_lock 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74815 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_lock on pid 74815' 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74815 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_lock_pp 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_lock_pp' 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.279 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_lock_pp 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.282 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.283 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_lock_pp 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74833 2024-03-30T13:59:23.284 INFO:tasks.workunit.client.0.smithi102.stdout:test api_lock_pp on pid 74833 2024-03-30T13:59:23.288 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_lock_pp on pid 74833' 2024-03-30T13:59:23.288 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74833 2024-03-30T13:59:23.288 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.288 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.288 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.288 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_misc 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.289 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_misc' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74763/exe 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.292 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.293 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.294 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.295 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.296 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_misc 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-30T13:59:23.298 INFO:tasks.workunit.client.0.smithi102.stdout:test api_misc on pid 74857 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_misc 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74857 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_misc on pid 74857' 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74857 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.301 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_misc_pp 2024-03-30T13:59:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74752/exe 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_misc_pp' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.305 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.306 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.307 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_misc_pp 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.309 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.310 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.310 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.310 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.310 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.310 INFO:tasks.workunit.client.0.smithi102.stdout:test api_misc_pp on pid 74875 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_misc_pp 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74875 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_misc_pp on pid 74875' 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74875 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.313 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.314 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_tier_pp 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_io.log 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_tier_pp' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.315 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74777/exe 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_tier_pp 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74795/exe 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.316 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.319 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_list: /' 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_tier_pp 2024-03-30T13:59:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_asio: /' 2024-03-30T13:59:23.321 INFO:tasks.workunit.client.0.smithi102.stdout:test api_tier_pp on pid 74897 2024-03-30T13:59:23.322 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74897 2024-03-30T13:59:23.322 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_tier_pp on pid 74897' 2024-03-30T13:59:23.322 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74897 2024-03-30T13:59:23.322 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.322 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.322 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_list.log 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.323 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_pool 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_pool' 2024-03-30T13:59:23.324 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_io: /' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_pool 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.325 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.327 INFO:tasks.workunit.client.0.smithi102.stdout:test api_pool on pid 74908 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_pool 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74908 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_pool on pid 74908' 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74908 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.330 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_snapshots 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_snapshots' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.331 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_snapshots 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74815/exe 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.333 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.334 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_snapshots 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-30T13:59:23.335 INFO:tasks.workunit.client.0.smithi102.stdout:test api_snapshots on pid 74923 2024-03-30T13:59:23.337 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74923 2024-03-30T13:59:23.337 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_snapshots on pid 74923' 2024-03-30T13:59:23.337 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74923 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_lock: /' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.338 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_snapshots_pp 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_snapshots_pp' 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.339 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_snapshots_pp 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-30T13:59:23.341 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.343 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.344 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.346 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.346 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74875/exe 2024-03-30T13:59:23.346 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.346 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_snapshots_pp 2024-03-30T13:59:23.346 INFO:tasks.workunit.client.0.smithi102.stdout:test api_snapshots_pp on pid 74938 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74938 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_snapshots_pp on pid 74938' 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74938 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.350 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.351 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.353 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.354 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_stat 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.355 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_stat' 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.357 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_stat 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.358 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.359 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.360 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74833/exe 2024-03-30T13:59:23.361 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.365 INFO:tasks.workunit.client.0.smithi102.stdout:test api_stat on pid 74987 2024-03-30T13:59:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_stat 2024-03-30T13:59:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74987 2024-03-30T13:59:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_stat on pid 74987' 2024-03-30T13:59:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=74987 2024-03-30T13:59:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_stat_pp 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_stat_pp' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.369 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.371 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.372 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.373 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.375 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.375 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_stat_pp 2024-03-30T13:59:23.375 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_stat_pp 2024-03-30T13:59:23.375 INFO:tasks.workunit.client.0.smithi102.stdout:test api_stat_pp on pid 75037 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75037 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_stat_pp on pid 75037' 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75037 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_watch_notify 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_watch_notify' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.381 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.382 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.383 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.385 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.386 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.387 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_watch_notify 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.390 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.391 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74938/exe 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.392 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74923/exe 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.393 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_watch_notify 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.395 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75090 2024-03-30T13:59:23.396 INFO:tasks.workunit.client.0.smithi102.stdout:test api_watch_notify on pid 75090 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_watch_notify on pid 75090' 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75090 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.403 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74857/exe 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74908/exe 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-30T13:59:23.404 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74897/exe 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_watch_notify_pp 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_watch_notify_pp' 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_misc: /' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-30T13:59:23.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_pool: /' 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.414 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_watch_notify_pp 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.415 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.416 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.417 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75037/exe 2024-03-30T13:59:23.418 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.419 INFO:tasks.workunit.client.0.smithi102.stdout:test api_watch_notify_pp on pid 75168 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_watch_notify_pp 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.423 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75168 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_watch_notify_pp on pid 75168' 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75168 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.424 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.425 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.426 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.427 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.428 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_cmd 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_cmd' 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_cmd 2024-03-30T13:59:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_cmd 2024-03-30T13:59:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/74987/exe 2024-03-30T13:59:23.431 INFO:tasks.workunit.client.0.smithi102.stdout:test api_cmd on pid 75219 2024-03-30T13:59:23.433 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75219 2024-03-30T13:59:23.433 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_cmd on pid 75219' 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75219 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_cmd_pp 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_cmd_pp' 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.434 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_cmd_pp 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.436 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_stat: /' 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.437 INFO:tasks.workunit.client.0.smithi102.stdout:test api_cmd_pp on pid 75243 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_cmd_pp 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.441 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75243 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_cmd_pp on pid 75243' 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75243 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_service 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_service' 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.445 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.446 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.447 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.448 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75090/exe 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.449 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.450 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_service 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-30T13:59:23.451 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_service 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75296 2024-03-30T13:59:23.454 INFO:tasks.workunit.client.0.smithi102.stdout:test api_service on pid 75296 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_service on pid 75296' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75296 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_service_pp 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_service_pp' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_service_pp 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.461 INFO:tasks.workunit.client.0.smithi102.stdout:test api_service_pp on pid 75323 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_service_pp 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75323 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_service_pp on pid 75323' 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75323 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_c_write_operations 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_c_write_operations' 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.465 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.466 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.467 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.468 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_c_write_operations 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75243/exe 2024-03-30T13:59:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75219/exe 2024-03-30T13:59:23.471 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.471 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.471 INFO:tasks.workunit.client.0.smithi102.stdout:test api_c_write_operations on pid 75361 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.473 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_c_write_operations 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75361 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_c_write_operations on pid 75361' 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75361 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_c_read_operations 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_c_read_operations' 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.475 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.479 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.480 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_c_read_operations 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.481 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.482 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.482 INFO:tasks.workunit.client.0.smithi102.stdout:test api_c_read_operations on pid 75384 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_c_read_operations 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75384 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_c_read_operations on pid 75384' 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75384 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s api_cls_remote_reads 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.485 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' api_cls_remote_reads' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_cmd: /' 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo api_cls_remote_reads 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.488 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.489 INFO:tasks.workunit.client.0.smithi102.stdout:test api_cls_remote_reads on pid 75402 2024-03-30T13:59:23.495 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.495 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.495 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.495 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=api_cls_remote_reads 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75402 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test api_cls_remote_reads on pid 75402' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75402 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75168/exe 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75296/exe 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.502 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s list_parallel 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' list_parallel' 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.504 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo list_parallel 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_service.log 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_service: /' 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75323/exe 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.511 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.512 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.513 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.514 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75361/exe 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.515 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.516 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-30T13:59:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75384/exe 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.519 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.520 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.521 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.522 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.523 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.523 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.523 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.523 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-30T13:59:23.523 INFO:tasks.workunit.client.0.smithi102.stdout:test list_parallel on pid 75497 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=list_parallel 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75497 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test list_parallel on pid 75497' 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75497 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.525 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.526 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.526 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.526 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75402/exe 2024-03-30T13:59:23.526 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s open_pools_parallel 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' open_pools_parallel' 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo open_pools_parallel 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.527 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.528 INFO:tasks.workunit.client.0.smithi102.stdout:test open_pools_parallel on pid 75527 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=open_pools_parallel 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75527 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test open_pools_parallel on pid 75527' 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75527 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s delete_pools_parallel 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' delete_pools_parallel' 2024-03-30T13:59:23.529 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.530 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo delete_pools_parallel 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.531 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-30T13:59:23.532 INFO:tasks.workunit.client.0.smithi102.stdout:test delete_pools_parallel on pid 75535 2024-03-30T13:59:23.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=delete_pools_parallel 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75535 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test delete_pools_parallel on pid 75535' 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75535 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.534 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s cls 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' cls' 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo cls 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.535 INFO:tasks.workunit.client.0.smithi102.stdout:test cls on pid 75555 2024-03-30T13:59:23.538 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=cls 2024-03-30T13:59:23.538 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75555 2024-03-30T13:59:23.538 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test cls on pid 75555' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75555 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s cmd 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' cmd' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.539 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.540 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.541 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.542 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.543 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo cmd 2024-03-30T13:59:23.544 INFO:tasks.workunit.client.0.smithi102.stdout:test cmd on pid 75575 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=cmd 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75575 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test cmd on pid 75575' 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75575 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s handler_error 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' handler_error' 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo handler_error 2024-03-30T13:59:23.550 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.551 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.552 INFO:tasks.workunit.client.0.smithi102.stdout:test handler_error on pid 75636 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=handler_error 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75636 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test handler_error on pid 75636' 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75636 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s io 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.556 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.557 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' io' 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.559 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.560 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.561 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.562 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75497/exe 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo io 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.563 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.565 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.565 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.565 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.565 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.565 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.565 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-30T13:59:23.565 INFO:tasks.workunit.client.0.smithi102.stdout:test io on pid 75682 2024-03-30T13:59:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=io 2024-03-30T13:59:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75682 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test io on pid 75682' 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75682 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ list_parallel: /' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s ec_io 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' ec_io' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.573 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75555/exe 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_cls 2024-03-30T13:59:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.580 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo ec_io 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75535/exe 2024-03-30T13:59:23.581 INFO:tasks.workunit.client.0.smithi102.stdout:test ec_io on pid 75710 2024-03-30T13:59:23.584 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=ec_io 2024-03-30T13:59:23.584 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75710 2024-03-30T13:59:23.584 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test ec_io on pid 75710' 2024-03-30T13:59:23.584 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75710 2024-03-30T13:59:23.584 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.584 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.585 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.586 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_cls.log 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s list 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ cls: /' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' list' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.587 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.588 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.589 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.590 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.591 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-30T13:59:23.592 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo list 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=list 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75575/exe 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75745 2024-03-30T13:59:23.594 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test list on pid 75745' 2024-03-30T13:59:23.595 INFO:tasks.workunit.client.0.smithi102.stdout:test list on pid 75745 2024-03-30T13:59:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75745 2024-03-30T13:59:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s ec_list 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' ec_list' 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo ec_list 2024-03-30T13:59:23.600 INFO:tasks.workunit.client.0.smithi102.stdout:test ec_list on pid 75763 2024-03-30T13:59:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=ec_list 2024-03-30T13:59:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75763 2024-03-30T13:59:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test ec_list on pid 75763' 2024-03-30T13:59:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75763 2024-03-30T13:59:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s misc 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' misc' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ cmd: /' 2024-03-30T13:59:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75527/exe 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_cmd 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.611 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.612 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.613 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.614 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo misc 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.615 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=misc 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75805 2024-03-30T13:59:23.616 INFO:tasks.workunit.client.0.smithi102.stdout:test misc on pid 75805 2024-03-30T13:59:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test misc on pid 75805' 2024-03-30T13:59:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75805 2024-03-30T13:59:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.620 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s pool 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' pool' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo pool 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.622 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.623 INFO:tasks.workunit.client.0.smithi102.stdout:test pool on pid 75831 2024-03-30T13:59:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=pool 2024-03-30T13:59:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75831 2024-03-30T13:59:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test pool on pid 75831' 2024-03-30T13:59:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75831 2024-03-30T13:59:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s read_operations 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' read_operations' 2024-03-30T13:59:23.626 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.627 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75636/exe 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo read_operations 2024-03-30T13:59:23.628 INFO:tasks.workunit.client.0.smithi102.stdout:test read_operations on pid 75848 2024-03-30T13:59:23.634 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=read_operations 2024-03-30T13:59:23.634 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75848 2024-03-30T13:59:23.634 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test read_operations on pid 75848' 2024-03-30T13:59:23.634 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75848 2024-03-30T13:59:23.634 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s snapshots 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' snapshots' 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.635 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ handler_error: /' 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.637 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.638 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_handler_error 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.639 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.640 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.641 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75682/exe 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo snapshots 2024-03-30T13:59:23.642 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.643 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=snapshots 2024-03-30T13:59:23.643 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75877 2024-03-30T13:59:23.643 INFO:tasks.workunit.client.0.smithi102.stdout:test snapshots on pid 75877 2024-03-30T13:59:23.646 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test snapshots on pid 75877' 2024-03-30T13:59:23.646 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75877 2024-03-30T13:59:23.646 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.646 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.646 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s watch_notify 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' watch_notify' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.647 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.648 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.649 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.650 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.651 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.652 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.653 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.654 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo watch_notify 2024-03-30T13:59:23.655 INFO:tasks.workunit.client.0.smithi102.stdout:test watch_notify on pid 75901 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=watch_notify 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75901 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test watch_notify on pid 75901' 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75901 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.662 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:++ printf %25s write_operations 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+ r=' write_operations' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.663 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ io: /' 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.664 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.665 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.667 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.668 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75763/exe 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75745/exe 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_io 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++ awk '{print $1}' 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++ echo write_operations 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.670 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.671 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.672 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.674 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_ec_list 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ ec_list: /' 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_list 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.675 INFO:tasks.workunit.client.0.smithi102.stdout:test write_operations on pid 75946 2024-03-30T13:59:23.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ ff=write_operations 2024-03-30T13:59:23.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75946 2024-03-30T13:59:23.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo 'test write_operations on pid 75946' 2024-03-30T13:59:23.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ pids[$f]=75946 2024-03-30T13:59:23.688 INFO:tasks.workunit.client.0.smithi102.stderr:+ ret=0 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74897 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74897 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_list.log 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.689 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ list: /' 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_io.log 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.693 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.695 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.696 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75831/exe 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75710/exe 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+ . /etc/bashrc 2024-03-30T13:59:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.698 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -z '' ']' 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:++ BASHRCSOURCED=Y 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:++ shopt -q login_shell 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_ec_io 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ umask 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.700 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:++ SHELL=/bin/bash 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_pool 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T13:59:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_pool.log 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ pool: /' 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.705 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75805/exe 2024-03-30T13:59:23.706 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ ec_io: /' 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T13:59:23.707 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:+++ return 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export S_COLORS=auto 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:+++ S_COLORS=auto 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.708 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_misc 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_misc.log 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ misc: /' 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75877/exe 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75848/exe 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.711 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.712 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.713 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ snapshots: /' 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_snapshots 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_read_operations 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T13:59:23.714 INFO:tasks.workunit.client.0.smithi102.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup 2024-03-30T13:59:23.715 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ read_operations: /' 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/locale 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T13:59:23.716 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:++++ LANG=en_US.UTF-8 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ LANG=en_US.UTF-8 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ unset LANG_backup config 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n '' ']' 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T13:59:23.718 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' dumb = linux ']' 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/less.sh 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ '[' '' ']' 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75946/exe 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:+++++ readlink /proc/75901/exe 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:++++ basename /usr/bin/bash 2024-03-30T13:59:23.719 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_declare='declare -f' 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+++ which_opt=-f 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export which_declare 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+++ export -f which 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset i 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:++ unset -f pathmunge 2024-03-30T13:59:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ export PATH 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ unset rc 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_write_operations 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ write_operations: /' 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph_test_neorados_watch_notify 2024-03-30T13:59:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ sed 's/^/ watch_notify: /' 2024-03-30T13:59:24.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:24 smithi138 ceph-mon[33397]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3969968593' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-30T13:59:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[29392]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3969968593' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-30T13:59:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[37717]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T13:59:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3969968593' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-30T13:59:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:25.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:59:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:59:26.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:59:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:59:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:29.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:29.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:29.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:29.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:29.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:29.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2177718536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4134451998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3362719461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T13:59:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2177718536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4134451998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3362719461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2177718536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4134451998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3362719461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:30.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T13:59:30.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/143777919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/26955587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2240804944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2539734865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/975428824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/115608240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2907898736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3366131961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2467166104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:32.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:32.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/143777919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/26955587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2240804944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2539734865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/975428824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/115608240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2907898736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3366131961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2467166104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi102-75994-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/143777919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/26955587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2240804944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2539734865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/975428824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/115608240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2907898736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3366131961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2467166104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:32.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:32.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi102-74805-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: pgmap v64: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: pgmap v65: 228 pgs: 77 creating+peering, 132 active+clean, 19 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: pgmap v66: 228 pgs: 77 creating+peering, 132 active+clean, 19 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: pgmap v68: 900 pgs: 679 creating+peering, 132 active+clean, 89 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3758748179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1392628517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi102-75092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1502653108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi102-75165-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:33.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1507180342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3203717506' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: pgmap v71: 1252 pgs: 679 creating+peering, 132 active+clean, 441 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1383948225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75364-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3153396891' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:33.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: pgmap v64: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:33.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: pgmap v65: 228 pgs: 77 creating+peering, 132 active+clean, 19 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: pgmap v66: 228 pgs: 77 creating+peering, 132 active+clean, 19 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: pgmap v68: 900 pgs: 679 creating+peering, 132 active+clean, 89 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3758748179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1392628517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi102-75092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1502653108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi102-75165-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: pgmap v64: 228 pgs: 96 unknown, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: pgmap v65: 228 pgs: 77 creating+peering, 132 active+clean, 19 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: pgmap v66: 228 pgs: 77 creating+peering, 132 active+clean, 19 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: pgmap v68: 900 pgs: 679 creating+peering, 132 active+clean, 89 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:33.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3758748179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1392628517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi102-75092-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1502653108' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi102-75165-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:33.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1507180342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3203717506' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: pgmap v71: 1252 pgs: 679 creating+peering, 132 active+clean, 441 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1383948225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75364-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3153396891' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:33.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi102-74920-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi102-74850-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi102-75928-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1507180342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3203717506' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: pgmap v71: 1252 pgs: 679 creating+peering, 132 active+clean, 441 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T13:59:33.443 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1383948225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75364-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3153396891' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [----------] Global test environment set-up. 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (6 ms) 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (4 ms) 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (5 ms) 2024-03-30T13:59:33.580 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (16 ms) 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (12 ms) 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (5 ms) 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (7 ms) 2024-03-30T13:59:33.581 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (6 ms) 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [----------] 12 tests from AsioRados (66 ms total) 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [----------] Global test environment tear-down 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (10261 ms total) 2024-03-30T13:59:33.582 INFO:tasks.workunit.client.0.smithi102.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-30T13:59:33.831 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] Global test environment set-up. 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosList.ListObjects (388 ms) 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (27 ms) 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset foo1,foo2,foo3 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo1 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo2 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo3 2024-03-30T13:59:33.832 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset foo1,foo4,foo5 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo4 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo5 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo1 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset foo6,foo7 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo7 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo6 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns1:foo4 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns1:foo5 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns2:foo7 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns2:foo6 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns1:foo1 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: :foo1 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: :foo2 2024-03-30T13:59:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: :foo3 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (73 ms) 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 1 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 10 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 13 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 7 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 14 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 0 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 15 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 11 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 5 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 8 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 6 0 2024-03-30T13:59:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 3 0 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 4 0 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 12 0 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 9 0 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 2 0 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (68 ms) 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: x cursor=MIN 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2024-03-30T13:59:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-03-30T13:59:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2024-03-30T13:59:33.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:566253c9:::13:head 2024-03-30T13:59:33.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-03-30T13:59:33.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2024-03-30T13:59:33.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-03-30T13:59:33.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=14 cursor=13:62a1935d:::14:head 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:863748b0:::15:head 2024-03-30T13:59:33.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-03-30T13:59:33.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:e960b815:::9:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:566253c9:::13:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2024-03-30T13:59:33.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=13 expected=13 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:863748b0:::15:head 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=15 expected=15 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=4 expected=4 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=3 expected=3 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-03-30T13:59:33.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=7 expected=7 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=2 expected=2 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:e960b815:::9:head 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=9 expected=9 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2024-03-30T13:59:33.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-03-30T13:59:34.553 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : ent list_parallel: process_1_[75720]: starting. 2024-03-30T13:59:34.553 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_1_[75720]: creating pool ceph_test_rados_list_parallel.smithi102-75678 2024-03-30T13:59:34.553 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_1_[75720]: created object 0... 2024-03-30T13:59:34.553 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_1_[75720]: created object 25... 2024-03-30T13:59:34.553 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_1_[75720]: created object 49... 2024-03-30T13:59:34.553 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_1_[75720]: finishing. 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_1_[75720]: shutting down. 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_2_[75721]: starting. 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_2_[75721]: listing objects. 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_2_[75721]: listed object 0... 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_2_[75721]: listed object 25... 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_2_[75721]: saw 50 objects 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_2_[75721]: shutting down. 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_3_[76771]: starting. 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_3_[76771]: creating pool ceph_test_rados_list_parallel.smithi102-75678 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_3_[76771]: created object 0... 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_3_[76771]: created object 25... 2024-03-30T13:59:34.554 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_3_[76771]: created object 49... 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_3_[76771]: finishing. 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_3_[76771]: shutting down. 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_4_[76772]: starting. 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_4_[76772]: listing objects. 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_4_[76772]: listed object 0... 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_4_[76772]: listed object 25... 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_4_[76772]: saw 50 objects 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_4_[76772]: shutting down. 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_5_[76773]: starting. 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_5_[76773]: removed 25 objects... 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_5_[76773]: removed half of the objects 2024-03-30T13:59:34.555 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_5_[76773]: removed 50 objects... 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_5_[76773]: removed 50 objects 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_5_[76773]: shutting down. 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_6_[76866]: starting. 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_6_[76866]: creating pool ceph_test_rados_list_parallel.smithi102-75678 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_6_[76866]: created object 0... 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_6_[76866]: created object 25... 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_6_[76866]: created object 49... 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_6_[76866]: finishing. 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_6_[76866]: shutting down. 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_7_[76867]: starting. 2024-03-30T13:59:34.556 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_7_[76867]: listing objects. 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_7_[76867]: listed object 0... 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_7_[76867]: listed object 25... 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_7_[76867]: listed object 50... 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_7_[76867]: saw 56 objects 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_7_[76867]: shutting down. 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_8_[76868]: starting. 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_8_[76868]: added 25 objects... 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_8_[76868]: added half of the objects 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_8_[76868]: added 50 objects... 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_8_[76868]: added 50 objects 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_8_[76868]: shutting down. 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.557 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_9_[76914]: starting. 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_9_[76914]: creating pool ceph_test_rados_list_parallel.smithi102-75678 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_9_[76914]: created object 0... 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_9_[76914]: created object 25... 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_9_[76914]: created object 49... 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_9_[76914]: finishing. 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_9_[76914]: shutting down. 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.754 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: starting. 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: listing objects. 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: listed object 0... 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: listed object 25... 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: listed object 50... 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: listed object 75... 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: saw 97 objects 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_10_[76915]: shutting down. 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_13_[76918]: starting. 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_13_[76918]: removed 25 objects... 2024-03-30T13:59:34.755 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_13_[76918]: removed half of the objects 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_13_[76918]: removed 50 objects... 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_13_[76918]: removed 50 objects 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_13_[76918]: shutting down. 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_11_[76916]: starting. 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_11_[76916]: added 25 objects... 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_11_[76916]: added half of the objects 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_11_[76916]: added 50 objects... 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_11_[76916]: added 50 objects 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_11_[76916]: shutting down. 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.756 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_12_[76917]: starting. 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_12_[76917]: added 25 objects... 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_12_[76917]: added half of the objects 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_12_[76917]: added 50 objects... 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_12_[76917]: added 50 objects 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_12_[76917]: shutting down. 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_14_[77003]: starting. 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_14_[77003]: creating pool ceph_test_rados_list_parallel.smithi102-75678 2024-03-30T13:59:34.757 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_14_[77003]: created object 0... 2024-03-30T13:59:34.758 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_14_[77003]: created object 25... 2024-03-30T13:59:34.758 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_14_[77003]: created object 49... 2024-03-30T13:59:34.758 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_14_[77003]: finishing. 2024-03-30T13:59:34.758 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_14_[77003]: shutting down. 2024-03-30T13:59:34.758 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.758 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.759 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.759 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:34.759 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: starting. 2024-03-30T13:59:34.759 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: listing objects. 2024-03-30T13:59:34.759 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: listed object 0... 2024-03-30T13:59:35.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3758748179' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1392628517' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi102-75092-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1502653108' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi102-75165-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1383948225' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75364-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:35.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3758748179' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1392628517' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi102-75092-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1502653108' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi102-75165-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1383948225' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75364-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:35.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3758748179' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1392628517' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi102-75092-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi102-74889-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1502653108' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi102-75165-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75362-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi102-75270-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi102-75095-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi102-75117-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi102-74869-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi102-74991-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi102-74825-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1383948225' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75364-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75524-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:35.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi102-75474-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi102-75232-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:35.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:35.684 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:59:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:59:35.723 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: listed object 2 api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-30T13:59:35.723 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-30T13:59:35.723 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-30T13:59:35.723 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-30T13:59:35.723 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (12209 ms) 2024-03-30T13:59:35.724 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (12209 ms total) 2024-03-30T13:59:35.724 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: 2024-03-30T13:59:35.724 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-30T13:59:35.724 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (12210 ms total) 2024-03-30T13:59:35.724 INFO:tasks.workunit.client.0.smithi102.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-30T13:59:36.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:59:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:59:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:37.068 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: Running main() from gmock_main.cc 2024-03-30T13:59:37.068 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [----------] Global test environment set-up. 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (13475 ms) 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [----------] 1 test from neocls_handler_error (13475 ms total) 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [----------] Global test environment tear-down 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [==========] 1 test from 1 test suite ran. (13475 ms total) 2024-03-30T13:59:37.069 INFO:tasks.workunit.client.0.smithi102.stdout: handler_error: [ PASSED ] 1 test. 2024-03-30T13:59:37.087 INFO:tasks.workunit.client.0.smithi102.stdout:5... 2024-03-30T13:59:37.087 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: listed object 50... 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: listed object 75... 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: listed object 100... 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: listed object 125... 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: saw 150 objects 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_15_[77004]: shutting down. 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_16_[77005]: starting. 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_16_[77005]: added 25 objects... 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_16_[77005]: added half of the objects 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_16_[77005]: added 50 objects... 2024-03-30T13:59:37.088 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_16_[77005]: added 50 objects 2024-03-30T13:59:37.089 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: process_16_[77005]: shutting down. 2024-03-30T13:59:37.089 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.089 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.089 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.089 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.089 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******************************* 2024-03-30T13:59:37.089 INFO:tasks.workunit.client.0.smithi102.stdout: list_parallel: ******* SUCCESS ********** 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (49 ms) 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (44 ms) 2024-03-30T13:59:37.092 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-30T13:59:37.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (13574 ms) 2024-03-30T13:59:37.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (13667 ms total) 2024-03-30T13:59:37.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: 2024-03-30T13:59:37.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-30T13:59:37.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (13667 ms total) 2024-03-30T13:59:37.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-30T13:59:37.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: pgmap v73: 1124 pgs: 560 creating+peering, 132 active+clean, 432 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/30792257' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:37.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: pgmap v73: 1124 pgs: 560 creating+peering, 132 active+clean, 432 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:59:37.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:37.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:37.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T13:59:37.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/30792257' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T13:59:37.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:37.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: pgmap v73: 1124 pgs: 560 creating+peering, 132 active+clean, 432 unknown; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi102-75994-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi102-75928-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/30792257' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:37.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi102-75421-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:38.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-30T13:59:38.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-30T13:59:38.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-30T13:59:38.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-30T13:59:38.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-30T13:59:38.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (52 ms) 2024-03-30T13:59:38.093 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (38 ms) 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (13531 ms) 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:35.731239+0000 mon.c [INF] from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:35.734866+0000 mon.c [INF] from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:35.739746+0000 mon.c [INF] from='client.? 172.21.15.102:0/492103904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:35.829141+0000 mon.c [INF] from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.012655+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.013314+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.013914+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.014411+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T13:59:38.094 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.014924+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T13:59:38.095 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.015448+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T13:59:38.095 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.015969+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T13:59:38.095 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.016482+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T13:59:38.095 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.016955+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T13:59:38.095 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:36.017405+0000 mon.b [INF] from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T13:59:38.155 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.045504+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-d api_c_read_operations: Running main() from gmock_main.cc 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (1 ms) 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (1321 ms) 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (17 ms) 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (6 ms) 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-30T13:59:38.156 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (1370 ms) 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (13 ms) 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (44 ms) 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (19 ms) 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3 ms) 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (3 ms) 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-30T13:59:38.157 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (3 ms) 2024-03-30T13:59:38.158 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-30T13:59:38.158 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (3 ms) 2024-03-30T13:59:38.158 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-30T13:59:38.158 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (7 ms) 2024-03-30T13:59:38.158 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-30T13:59:38.158 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (7 ms) 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (7 ms) 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (2838 ms total) 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (14680 ms total) 2024-03-30T13:59:38.159 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/492103904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T13:59:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: 11.7 deep-scrub starts 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: 11.7 deep-scrub ok 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: 11.4 deep-scrub starts 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: 11.4 deep-scrub ok 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:38.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:38.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:38.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/492103904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T13:59:38.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: 11.7 deep-scrub starts 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: 11.7 deep-scrub ok 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: 11.4 deep-scrub starts 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: 11.4 deep-scrub ok 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:38.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/492103904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T13:59:38.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: 11.7 deep-scrub starts 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: 11.7 deep-scrub ok 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: 11.4 deep-scrub starts 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: 11.4 deep-scrub ok 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:38.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.134 INFO:tasks.workunit.client.0.smithi102.stdout:omain=osd"]}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.053164+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.054743+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.055377+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.073238+0000 mon.b [INF] from='client.? 172.21.15.102:0/2384221459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.073710+0000 mon.b [INF] from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.081624+0000 mon.b [INF] from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:37.834911+0000 mon.c [INF] from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.087462+0000 mon.a [INF] from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.087852+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.095071+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.095274+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.135 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.096456+0000 mon.b [INF] from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.136 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.096975+0000 mon.c [INF] from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:39.136 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.097889+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:39.136 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.097978+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:39.136 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.098066+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:39.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:39.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T13:59:39.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.0 deep-scrub starts 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.0 deep-scrub ok 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: pgmap v76: 852 pgs: 104 creating+peering, 644 active+clean, 104 unknown; 610 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 135 KiB/s wr, 684 op/s 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.9 deep-scrub starts 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.9 deep-scrub ok 2024-03-30T13:59:39.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.3 deep-scrub starts 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.3 deep-scrub ok 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.2 deep-scrub starts 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.6 deep-scrub starts 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.6 deep-scrub ok 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.2 deep-scrub ok 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2384221459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.8 deep-scrub starts 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.8 deep-scrub ok 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.5 deep-scrub starts 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: 11.5 deep-scrub ok 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:39.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2294822935' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: onexx 2024-03-30T13:59:39.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:39.438 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T13:59:39.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.0 deep-scrub starts 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.0 deep-scrub ok 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: pgmap v76: 852 pgs: 104 creating+peering, 644 active+clean, 104 unknown; 610 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 135 KiB/s wr, 684 op/s 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.9 deep-scrub starts 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.9 deep-scrub ok 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.3 deep-scrub starts 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.3 deep-scrub ok 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.2 deep-scrub starts 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.6 deep-scrub starts 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.6 deep-scrub ok 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.2 deep-scrub ok 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2384221459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.8 deep-scrub starts 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.8 deep-scrub ok 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.5 deep-scrub starts 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: 11.5 deep-scrub ok 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:39.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2294822935' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: onexx 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.442 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T13:59:39.443 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.0 deep-scrub starts 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.0 deep-scrub ok 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: pgmap v76: 852 pgs: 104 creating+peering, 644 active+clean, 104 unknown; 610 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 135 KiB/s wr, 684 op/s 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.9 deep-scrub starts 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.9 deep-scrub ok 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.3 deep-scrub starts 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.3 deep-scrub ok 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.2 deep-scrub starts 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.6 deep-scrub starts 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.6 deep-scrub ok 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.2 deep-scrub ok 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2384221459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.8 deep-scrub starts 2024-03-30T13:59:39.444 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.8 deep-scrub ok 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.5 deep-scrub starts 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: 11.5 deep-scrub ok 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:39.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:39.446 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:39.447 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:39.447 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:39.447 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2294822935' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.447 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: onexx 2024-03-30T13:59:39.447 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T13:59:39.447 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:39.448 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:40.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: 11.1 deep-scrub starts 2024-03-30T13:59:40.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: 11.1 deep-scrub ok 2024-03-30T13:59:40.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:40.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-30T13:59:40.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T13:59:40.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: twoxx 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:40.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T13:59:40.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:40.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:40.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi102-75165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:40.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: 11.1 deep-scrub starts 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: 11.1 deep-scrub ok 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: twoxx 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T13:59:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi102-75165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: 11.1 deep-scrub starts 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: 11.1 deep-scrub ok 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: twoxx 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:40.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi102-74805-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]': finished 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T13:59:40.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:40.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:40.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi102-75165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:40.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: a delete_pools_parallel: process_1_[75795]: starting. 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_1_[75795]: creating pool ceph_test_rados_delete_pools_parallel.smithi102-75713 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_1_[75795]: created object 0... 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_1_[75795]: created object 25... 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_1_[75795]: created object 49... 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_1_[75795]: finishing. 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_1_[75795]: shutting down. 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_2_[75797]: starting. 2024-03-30T13:59:41.575 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_2_[75797]: deleting pool ceph_test_rados_delete_pools_parallel.smithi102-75713 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_2_[75797]: shutting down. 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: ******************************* 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_3_[77153]: starting. 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_3_[77153]: creating pool ceph_test_rados_delete_pools_parallel.smithi102-75713 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_3_[77153]: created object 0... 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_3_[77153]: created object 25... 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_3_[77153]: created object 49... 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_3_[77153]: finishing. 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_3_[77153]: shutting down. 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: ******************************* 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_5_[77155]: starting. 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_5_[77155]: listing objects. 2024-03-30T13:59:41.576 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_5_[77155]: listed object 0... 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_5_[77155]: listed object 25... 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_5_[77155]: saw 50 objects 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_5_[77155]: shutting down. 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: ******************************* 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_4_[77154]: starting. 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_4_[77154]: deleting pool ceph_test_rados_delete_pools_parallel.smithi102-75713 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: process_4_[77154]: shutting down. 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: ******************************* 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: ******************************* 2024-03-30T13:59:41.577 INFO:tasks.workunit.client.0.smithi102.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: Running main() from gmock_main.cc 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [----------] Global test environment set-up. 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [ OK ] NeoRadosCls.DNE (12179 ms) 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (5859 ms) 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [----------] 2 tests from NeoRadosCls (18038 ms total) 2024-03-30T13:59:41.578 INFO:tasks.workunit.client.0.smithi102.stdout: cls: 2024-03-30T13:59:41.579 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [----------] Global test environment tear-down 2024-03-30T13:59:41.579 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [==========] 2 tests from 1 test suite ran. (18039 ms total) 2024-03-30T13:59:41.579 INFO:tasks.workunit.client.0.smithi102.stdout: cls: [ PASSED ] 2 tests. 2024-03-30T13:59:41.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: pgmap v79: 852 pgs: 53 creating+peering, 471 active+clean, 328 unknown; 587 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 23 KiB/s wr, 545 op/s 2024-03-30T13:59:41.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:41.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T13:59:41.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:41.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:41.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:41.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T13:59:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:41.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:41.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:41.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:41.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: pgmap v79: 852 pgs: 53 creating+peering, 471 active+clean, 328 unknown; 587 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 23 KiB/s wr, 545 op/s 2024-03-30T13:59:41.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:41.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T13:59:41.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:41.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:41.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:41.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:41.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:41.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T13:59:41.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:41.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:41.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:41.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:41.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:41.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: pgmap v79: 852 pgs: 53 creating+peering, 471 active+clean, 328 unknown; 587 KiB data, 264 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 23 KiB/s wr, 545 op/s 2024-03-30T13:59:41.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:41.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T13:59:41.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:41.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:41.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:41.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:41.694 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:41.694 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T13:59:41.694 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:41.694 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:41.694 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:41.695 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.695 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:41.695 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:41.695 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:41.695 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:42.209 INFO:tasks.workunit.client.0.smithi102.stdout:pi_cmd: got: 2024-03-30T13:59:38.105171+0000 mon.c [INF] from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:42.209 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.105923+0000 mon.c [INF] from='client.? 172.21.15.102:0/1377128481' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:42.209 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.106844+0000 mon.b [INF] from='client.? 172.21.15.102:0/2294822935' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.209 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.108808+0000 client.admin [INF] onexx 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.877437+0000 mon.a [INF] from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.877724+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.878144+0000 mon.a [INF] from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.878577+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.878627+0000 mon.b [INF] from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.883077+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.883099+0000 mon.c [INF] from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.883825+0000 mon.c [INF] from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.884294+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.884787+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:38.885281+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi102-75994-1"}]: dispatch 2024-03-30T13:59:42.210 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.206172+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.206558+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.206845+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.208863+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.215967+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi102-74850-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.216033+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi102-74869-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.222213+0000 mon.c [INF] from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.496370+0000 mon.c [INF] from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:42.339 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.496705+0000 mon.b [INF] from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.496916+0000 mon.b [INF] from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.761115+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.763814+0000 mon.a [INF] from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.764264+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.764307+0000 mon.c [INF] from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.769871+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:40.770377+0000 mon.c [INF] from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.340 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.265157+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.512 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.265443+0000 mon.a [INF] from='c api_io_pp: Running main() from gmock_main.cc 2024-03-30T13:59:42.512 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-30T13:59:42.512 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-30T13:59:42.512 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-30T13:59:42.512 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: seed 74850 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (33 ms) 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (9 ms) 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (3 ms) 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2 ms) 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-30T13:59:42.513 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (2 ms) 2024-03-30T13:59:42.514 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (7 ms) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (3 ms) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (5 ms) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (3 ms) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (101 ms total) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (2255 ms) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (26 ms) 2024-03-30T13:59:42.515 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-30T13:59:42.516 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-03-30T13:59:42.516 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-30T13:59:42.516 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5 ms) 2024-03-30T13:59:42.516 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-30T13:59:42.516 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] Li cmd: Running main() from gmock_main.cc 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [----------] Global test environment set-up. 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9638 ms) 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (3869 ms) 2024-03-30T13:59:42.651 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-30T13:59:42.652 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (5571 ms) 2024-03-30T13:59:42.652 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [----------] 3 tests from NeoRadosCmd (19078 ms total) 2024-03-30T13:59:42.652 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: 2024-03-30T13:59:42.652 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [----------] Global test environment tear-down 2024-03-30T13:59:42.652 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [==========] 3 tests from 1 test suite ran. (19078 ms total) 2024-03-30T13:59:42.652 INFO:tasks.workunit.client.0.smithi102.stdout: cmd: [ PASSED ] 3 tests. 2024-03-30T13:59:42.656 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_1_[75871]: starting. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_1_[75871]: creating pool ceph_test_rados_open_pools_parallel.smithi102-75779 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_1_[75871]: created object 0... 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_1_[75871]: created object 25... 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_1_[75871]: created object 49... 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_1_[75871]: finishing. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_1_[75871]: shutting down. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_2_[75872]: starting. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_2_[75872]: rados_pool_create. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_2_[75872]: rados_ioctx_create. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_2_[75872]: shutting down. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: ******************************* 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_3_[77162]: starting. 2024-03-30T13:59:42.657 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_3_[77162]: creating pool ceph_test_rados_open_pools_parallel.smithi102-75779 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_3_[77162]: created object 0... 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_3_[77162]: created object 25... 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_3_[77162]: created object 49... 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_3_[77162]: finishing. 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_3_[77162]: shutting down. 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: ******************************* 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_4_[77163]: starting. 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_4_[77163]: rados_pool_create. 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_4_[77163]: rados_ioctx_create. 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: process_4_[77163]: shutting down. 2024-03-30T13:59:42.658 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: ******************************* 2024-03-30T13:59:42.659 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: ******************************* 2024-03-30T13:59:42.659 INFO:tasks.workunit.client.0.smithi102.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: pgmap v82: 916 pgs: 1 creating+activating, 137 creating+peering, 630 active+clean, 1 active+clean+snaptrim_wait, 147 unknown; 145 MiB data, 687 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi102-75165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:42.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2975480563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:42.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: pgmap v82: 916 pgs: 1 creating+activating, 137 creating+peering, 630 active+clean, 1 active+clean+snaptrim_wait, 147 unknown; 145 MiB data, 687 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:59:42.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi102-75165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2975480563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: pgmap v82: 916 pgs: 1 creating+activating, 137 creating+peering, 630 active+clean, 1 active+clean+snaptrim_wait, 147 unknown; 145 MiB data, 687 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi102-75165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2975480563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:42.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout:lient.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.270235+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.270579+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.270760+0000 mon.a [INF] from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi102-75165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.270880+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.270940+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi102-75994-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.276099+0000 mon.b [INF] from='client.? 172.21.15.102:0/458660040' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:43.415 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.276697+0000 mon.c [INF] from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.278982+0000 mon.c [INF] from='client.? 172.21.15.102:0/2975480563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.575974+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.576219+0000 mon.c [INF] from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.578157+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.578495+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.579684+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:41.592728+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.214606+0000 mon.c [INF] from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.338192+0000 client.admin [INF] threexx 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.338632+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.340933+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:43.416 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341080+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341164+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341241+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341313+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341384+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341451+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341491+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.341546+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.348162+0000 mon.b [INF] from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.354691+0000 mon.c [INF] from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:43.417 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.354878+0000 mon.c [INF] from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:43.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: threexx 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:43.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:43.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:43.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:43.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:43.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: threexx 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: threexx 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi102-74869-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi102-75232-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi102-74920-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi102-74825-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi102-75928-1"}]': finished 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:43.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:43.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:43.693 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:44 smithi138 ceph-mon[33397]: pgmap v84: 860 pgs: 1 active+clean+snaptrim, 1 creating+activating, 141 creating+peering, 548 active+clean, 1 active+clean+snaptrim_wait, 168 unknown; 145 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 36 MiB/s wr, 5 op/s 2024-03-30T13:59:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:44.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:44 smithi102 ceph-mon[29392]: pgmap v84: 860 pgs: 1 active+clean+snaptrim, 1 creating+activating, 141 creating+peering, 548 active+clean, 1 active+clean+snaptrim_wait, 168 unknown; 145 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 36 MiB/s wr, 5 op/s 2024-03-30T13:59:44.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:44 smithi102 ceph-mon[37717]: pgmap v84: 860 pgs: 1 active+clean+snaptrim, 1 creating+activating, 141 creating+peering, 548 active+clean, 1 active+clean+snaptrim_wait, 168 unknown; 145 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 36 MiB/s wr, 5 op/s 2024-03-30T13:59:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi102-75994-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:45.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:59:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-30T13:59:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: fourxx 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2"}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1071638503' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/709211577' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T13:59:45.687 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.688 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T13:59:45.854 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.357836+0000 mon.b [INF] from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "bRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-30T13:59:45.854 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-30T13:59:45.854 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5 ms) 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (3 ms) 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (7 ms) 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-30T13:59:45.855 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (3 ms) 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (7 ms) 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (2350 ms total) 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (22605 ms total) 2024-03-30T13:59:45.856 INFO:tasks.workunit.client.0.smithi102.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: handle_notify cookie 94202681273104 notify_id 330712481792 notifier_gid 24821 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (12049 ms) 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: Trying... 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: handle_notify cookie 94202693983296 notify_id 360777252864 notifier_gid 25093 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: Waiting for 3.000000000s 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: Timed out. 2024-03-30T13:59:45.857 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: Flushing... 2024-03-30T13:59:45.858 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: Flushed... 2024-03-30T13:59:45.858 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (10123 ms) 2024-03-30T13:59:45.858 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (22172 ms total) 2024-03-30T13:59:45.858 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: 2024-03-30T13:59:45.858 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-30T13:59:45.858 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (22172 ms total) 2024-03-30T13:59:45.858 INFO:tasks.workunit.client.0.smithi102.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-30T13:59:45.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:45.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-30T13:59:45.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:45.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: fourxx 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2"}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1071638503' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:45.941 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4140439091' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/901375096' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: fourxx 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2"}]: dispatch 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1071638503' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/709211577' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T13:59:45.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:45.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/709211577' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-4", "overlaypool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi102-75421-1"}]': finished 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:45.946 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi102-74805-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/22986548' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]: dispatch 2024-03-30T13:59:45.947 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: seed 75165 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (5 ms) 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (5 ms) 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-30T13:59:46.141 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (287 ms) 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (338 ms) 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (18 ms) 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (653 ms total) 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (917 ms) 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (158 ms) 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-30T13:59:46.142 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (249 ms) 2024-03-30T13:59:46.143 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-30T13:59:46.143 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7 ms) 2024-03-30T13:59:46.143 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1331 ms total) 2024-03-30T13:59:46.143 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: 2024-03-30T13:59:46.143 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-30T13:59:46.143 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (22774 ms total) 2024-03-30T13:59:46.143 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: Running main() from gmock_main.cc 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [----------] Global test environment set-up. 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [----------] 3 tests from NeoradosList 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [ OK ] NeoradosList.ListObjects (11143 ms) 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [ OK ] NeoradosList.ListObjectsNS (7593 ms) 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-30T13:59:46.910 INFO:tasks.workunit.client.0.smithi102.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4555 ms) 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: list: [----------] 3 tests from NeoradosList (23291 ms total) 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: list: 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: list: [----------] Global test environment tear-down 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: list: [==========] 3 tests from 1 test suite ran. (23291 ms total) 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: list: [ PASSED ] 3 tests. 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: Running main() from gmock_main.cc 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [----------] Global test environment set-up. 2024-03-30T13:59:46.911 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (348 ms) 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: no timeout :/ 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: no timeout :/ 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: no timeout :/ 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: no timeout :/ 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: no timeout :/ 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (588 ms) 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (7 ms) 2024-03-30T13:59:46.912 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3 ms) 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-30T13:59:46.913 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (2 ms) 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.RmXattr (7 ms) 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIo.XattrIter (4 ms) 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [----------] 14 tests from LibRadosIo (982 ms total) 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1148 ms) 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (7 ms) 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2024-03-30T13:59:46.914 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (5 ms) 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (7 ms) 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-30T13:59:46.915 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1195 ms total) 2024-03-30T13:59:46.916 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: 2024-03-30T13:59:46.916 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [----------] Global test environment tear-down 2024-03-30T13:59:46.916 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [==========] 24 tests from 2 test suites ran. (23659 ms total) 2024-03-30T13:59:46.916 INFO:tasks.workunit.client.0.smithi102.stdout: api_io: [ PASSED ] 24 tests. 2024-03-30T13:59:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: pgmap v87: 820 pgs: 1 active+clean+snaptrim, 38 creating+peering, 450 active+clean, 1 active+clean+snaptrim_wait, 330 unknown; 145 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 6 op/s 2024-03-30T13:59:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-30T13:59:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T13:59:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]': finished 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1702166153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:46.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T13:59:46.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:46.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:59:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: pgmap v87: 820 pgs: 1 active+clean+snaptrim, 38 creating+peering, 450 active+clean, 1 active+clean+snaptrim_wait, 330 unknown; 145 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 6 op/s 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:46.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]': finished 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1702166153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]: dispatch 2024-03-30T13:59:46.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:46.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T13:59:46.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:46.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T13:59:46.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:46.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: pgmap v87: 820 pgs: 1 active+clean+snaptrim, 38 creating+peering, 450 active+clean, 1 active+clean+snaptrim_wait, 330 unknown; 145 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 6 op/s 2024-03-30T13:59:46.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-30T13:59:46.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T13:59:46.942 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi102-75928-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3385869353' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi102-75165-7"}]': finished 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-4-cache", "mode": "writeback"}]': finished 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi102-74850-22"}]': finished 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1702166153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/699357988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]: dispatch 2024-03-30T13:59:46.943 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]: dispatch 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T13:59:46.944 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:47.587 INFO:tasks.workunit.client.0.smithi102.stdout:testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.358025+0000 mon.b [INF] from='client.? 172.21.15.102:0/1078390801' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.359063+0000 mon.b [INF] from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.431303+0000 mon.c [INF] from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.506509+0000 mon.c [INF] from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.652384+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi102-74920-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.653071+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi102-74991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.654035+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi102-75421-1"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.654243+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi102-75232-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.654664+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi102-74825-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:42.655141+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:43.151323+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi102-75928-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: got: 2024-03-30T13:59:43.156598+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi102-75107-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (10544 ms) 2024-03-30T13:59:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (24165 ms total) 2024-03-30T13:59:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: 2024-03-30T13:59:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-30T13:59:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (24166 ms total) 2024-03-30T13:59:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-30T13:59:47.956 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: Running main() from gmock_main.cc 2024-03-30T13:59:47.956 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [----------] Global test environment set-up. 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (15508 ms) 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (8809 ms) 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (24317 ms total) 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [----------] Global test environment tear-down 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (24317 ms total) 2024-03-30T13:59:47.957 INFO:tasks.workunit.client.0.smithi102.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]: dispatch 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3128394662' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:48.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]: dispatch 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3532159485' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:48.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T13:59:48.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]: dispatch 2024-03-30T13:59:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T13:59:48.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3128394662' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3532159485' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:48.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]: dispatch 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi102-74869-16"}]': finished 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3128394662' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2910364147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:48.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3755578700' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]: dispatch 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3532159485' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]: dispatch 2024-03-30T13:59:48.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T13:59:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: pgmap v91: 644 pgs: 18 creating+peering, 522 active+clean, 104 unknown; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-30T13:59:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]: dispatch 2024-03-30T13:59:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]': finished 2024-03-30T13:59:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3532159485' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]': finished 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]': finished 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3491507815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:49.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:49.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: pgmap v91: 644 pgs: 18 creating+peering, 522 active+clean, 104 unknown; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-30T13:59:49.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]: dispatch 2024-03-30T13:59:49.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]': finished 2024-03-30T13:59:49.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:49.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3532159485' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]': finished 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]': finished 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3491507815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:49.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: pgmap v91: 644 pgs: 18 creating+peering, 522 active+clean, 104 unknown; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]: dispatch 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-4"}]': finished 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi102-74805-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3532159485' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:49.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi102-75994-2"}]': finished 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi102-75107-1","app":"app1","key":"key1"}]': finished 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75421-6", "pg_num": 4}]': finished 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/194900190' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3491507815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:49.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:49.828 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: Running main() from gmock_main.cc 2024-03-30T13:59:49.828 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-30T13:59:49.828 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [----------] Global test environment set-up. 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStat.Stat (1333 ms) 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (1353 ms) 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStat.StatNS (23 ms) 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (6 ms) 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [----------] 5 tests from LibRadosStat (2716 ms total) 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: 2024-03-30T13:59:49.829 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1504 ms) 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (15 ms) 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1521 ms total) 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [----------] Global test environment tear-down 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (26416 ms total) 2024-03-30T13:59:49.830 INFO:tasks.workunit.client.0.smithi102.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: pgmap v94: 800 pgs: 12 creating+peering, 488 active+clean, 300 unknown; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4196653715' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:51.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: pool 'PoolQuotaPP_smithi102-74825-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-30T13:59:51.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:51 smithi138 ceph-mon[33397]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: pgmap v94: 800 pgs: 12 creating+peering, 488 active+clean, 300 unknown; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4196653715' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: pool 'PoolQuotaPP_smithi102-74825-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-30T13:59:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[37717]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: pgmap v94: 800 pgs: 12 creating+peering, 488 active+clean, 300 unknown; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-4", "tierpool": "test-rados-api-smithi102-75117-4-cache"}]': finished 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi102-75092-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1464457904' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:51.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4196653715' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: Health check update: 12 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:51.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: pool 'PoolQuotaPP_smithi102-74825-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-30T13:59:51.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:51 smithi102 ceph-mon[29392]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: Running main() from gmock_main.cc 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [----------] Global test environment set-up. 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ OK ] NeoRadosPools.PoolList (10211 ms) 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4250 ms) 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3462 ms) 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-30T13:59:52.433 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4285 ms) 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (3508 ms) 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (3065 ms) 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [----------] 6 tests from NeoRadosPools (28781 ms total) 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [----------] Global test environment tear-down 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [==========] 6 tests from 1 test suite ran. (28781 ms total) 2024-03-30T13:59:52.434 INFO:tasks.workunit.client.0.smithi102.stdout: pool: [ PASSED ] 6 tests. 2024-03-30T13:59:52.435 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-30T13:59:52.435 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-30T13:59:52.435 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-30T13:59:52.435 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-30T13:59:52.435 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: seed 74991 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (334 ms) 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (8 ms) 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1069 ms) 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1008 ms) 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (66 ms) 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-30T13:59:52.436 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (57 ms) 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (148 ms) 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2695 ms total) 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1471 ms) 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (39 ms) 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1020 ms) 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1081 ms) 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-30T13:59:52.437 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (14 ms) 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (10 ms) 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (7 ms) 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3646 ms total) 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (29115 ms total) 2024-03-30T13:59:52.438 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-30T13:59:52.463 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: Running main() from gmock_main.cc 2024-03-30T13:59:52.463 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-30T13:59:52.463 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [----------] Global test environment set-up. 2024-03-30T13:59:52.463 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-30T13:59:52.463 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-30T13:59:52.463 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (55 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.LockShared (3 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1259 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1125 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-03-30T13:59:52.464 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [----------] 8 tests from LibRadosLock (2458 ms total) 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1492 ms) 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (17 ms) 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1010 ms) 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1044 ms) 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (46 ms) 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (16 ms) 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-30T13:59:52.465 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (7 ms) 2024-03-30T13:59:52.466 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-30T13:59:52.466 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (7 ms) 2024-03-30T13:59:52.466 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3639 ms total) 2024-03-30T13:59:52.466 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: 2024-03-30T13:59:52.466 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [----------] Global test environment tear-down 2024-03-30T13:59:52.466 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (29174 ms total) 2024-03-30T13:59:52.466 INFO:tasks.workunit.client.0.smithi102.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]': finished 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]: dispatch 2024-03-30T13:59:52.665 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: pgmap v97: 624 pgs: 29 creating+peering, 566 active+clean, 29 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 3 op/s 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/343101516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2986948134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:52.666 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.667 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.667 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:59:52.667 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]': finished 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]: dispatch 2024-03-30T13:59:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: pgmap v97: 624 pgs: 29 creating+peering, 566 active+clean, 29 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 3 op/s 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/343101516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2986948134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:59:52.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi102-75270-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi102-75232-7"}]': finished 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "overlaypool": "test-rados-api-smithi102-75421-6"}]': finished 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]: dispatch 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1163523679' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2528789338' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:52.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2083096748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: pgmap v97: 624 pgs: 29 creating+peering, 566 active+clean, 29 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 3 op/s 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/343101516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2986948134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:52.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T13:59:52.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: pgmap v99: 784 pgs: 35 creating+peering, 584 active+clean, 165 unknown; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 4 op/s 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]': finished 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi102-75928-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: pgmap v99: 784 pgs: 35 creating+peering, 584 active+clean, 165 unknown; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 4 op/s 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]': finished 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi102-75928-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: pgmap v99: 784 pgs: 35 creating+peering, 584 active+clean, 165 unknown; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 4 op/s 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75421-6", "mode": "writeback"}]': finished 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi102-74991-10"}]': finished 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi102-75928-2"}]': finished 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi102-74920-10"}]': finished 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi102-74805-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:53.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi102-75928-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [----------] Global test environment set-up. 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5709 ms) 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ OK ] LibRadosService.RegisterLate (15 ms) 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: cluster: 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: id: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:59:54.844 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: health: HEALTH_WARN 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 3 pool(s) do not have an application enabled 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: services: 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: data: 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: pools: 9 pools, 228 pgs 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: pgs: 42.105% pgs unknown 2024-03-30T13:59:54.845 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 132 active+clean 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 96 unknown 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: io: 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: cluster: 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: id: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: health: HEALTH_WARN 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 32 pool(s) do not have an application enabled 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: services: 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-30T13:59:54.846 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: data: 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: pools: 41 pools, 1252 pgs 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: pgs: 35.224% pgs unknown 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 54.233% pgs not active 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 679 creating+peering 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 441 unknown 2024-03-30T13:59:54.847 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 132 active+clean 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: io: 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: client: 852 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5634 ms) 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ OK ] LibRadosService.Status (20025 ms) 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [----------] 4 tests from LibRadosService (31383 ms total) 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [----------] Global test environment tear-down 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31383 ms total) 2024-03-30T13:59:54.848 INFO:tasks.workunit.client.0.smithi102.stdout: api_service: [ PASSED ] 4 tests. 2024-03-30T13:59:54.877 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-30T13:59:54.877 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5044 ms) 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (37 ms) 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20854 ms) 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: attempt 0 of 20 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: failed to deregister: 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: { 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "epoch": 19, 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "modified": "2024-03-30T13:59:50.264300+0000", 2024-03-30T13:59:54.878 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "services": { 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "laundry": { 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "daemons": { 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "summary": "", 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "close-test-pid75449": { 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "start_epoch": 19, 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "start_stamp": "2024-03-30T13:59:49.812476+0000", 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "gid": 26465, 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "addr": "172.21.15.102:0/4049798006", 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "metadata": { 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "arch": "x86_64", 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_release": "squid", 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_version_short": "19.0.0-1967-g4e50bbcd", 2024-03-30T13:59:54.879 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro": "centos", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro_version": "9", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "foo": "bar", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "hostname": "smithi102", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "kernel_version": "5.14.0-434.el9.x86_64", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "mem_total_kb": "32488456", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "os": "Linux", 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "this": "that" 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: }, 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "task_status": {} 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: }, 2024-03-30T13:59:54.880 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "pid75410": { 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "start_epoch": 18, 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "start_stamp": "2024-03-30T13:59:34.839474+0000", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "gid": 25100, 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "addr": "172.21.15.102:0/2144259058", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "metadata": { 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "arch": "x86_64", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_release": "squid", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_version_short": "19.0.0-1967-g4e50bbcd", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro": "centos", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro_version": "9", 2024-03-30T13:59:54.881 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "foo": "bar", 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "hostname": "smithi102", 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024", 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "kernel_version": "5.14.0-434.el9.x86_64", 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "mem_total_kb": "32488456", 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "os": "Linux", 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "this": "that" 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: }, 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "task_status": {} 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: } 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: } 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: }, 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "rgw": { 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "daemons": { 2024-03-30T13:59:54.882 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "summary": "", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "24368": { 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "start_epoch": 12, 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "start_stamp": "2024-03-30T13:56:52.238357+0000", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "gid": 24368, 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "addr": "172.21.15.102:0/1642360733", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "metadata": { 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "arch": "x86_64", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_release": "squid", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "ceph_version_short": "19.0.0-1967-g4e50bbcd", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "container_hostname": "smithi102", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro": "centos", 2024-03-30T13:59:54.883 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "distro_version": "8", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "frontend_type#0": "beast", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "hostname": "smithi102", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "id": "foo.a", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "kernel_version": "5.14.0-434.el9.x86_64", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "mem_total_kb": "32488456", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "num_handles": "1", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "os": "Linux", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "pid": "2", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "realm_id": "", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "realm_name": "", 2024-03-30T13:59:54.884 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "zone_id": "2fd0b4ce-3bbc-440c-8319-8e09d0117317", 2024-03-30T13:59:54.885 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "zone_name": "default", 2024-03-30T13:59:54.885 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "zonegroup_id": "98cf89e7-acfb-493f-9f62-d4ca45ab2e31", 2024-03-30T13:59:55.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 13:59:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T13:59:55.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "zonegroup_name": "de api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-30T13:59:55.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-30T13:59:55.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-30T13:59:55.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-30T13:59:55.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: notify 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (3617 ms) 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (3622 ms total) 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: notify 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (89 ms) 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: notify 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5635 ms) 2024-03-30T13:59:55.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (26 ms) 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (21 ms) 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241787552 notify_id 408021893121 notifier_gid 25186 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (25 ms) 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241098224 notify_id 408021893121 notifier_gid 25186 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (40 ms) 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241787552 notify_id 408021893122 notifier_gid 25186 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (24 ms) 2024-03-30T13:59:55.904 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241098224 notify_id 408021893123 notifier_gid 25186 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (33 ms) 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241098224 notify_id 408021893124 notifier_gid 25186 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (40 ms) 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241098224 notify_id 408021893125 notifier_gid 25186 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (49 ms) 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: trying... 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241787552 notify_id 408021893126 notifier_gid 25186 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: timed out 2024-03-30T13:59:55.905 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: flushing 2024-03-30T13:59:56.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi102-75928-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:56.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:56 smithi138 ceph-mon[33397]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T13:59:56.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi102-75928-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T13:59:56.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2853160790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:56.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:56.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:56.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:56.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi102-75928-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:56.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[29392]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi102-75928-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2853160790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi102-75928-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[37717]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi102-75928-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2853160790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:56.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:56.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:56.854 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 13:59:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:13:59:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: pgmap v102: 520 pgs: 456 active+clean, 64 unknown; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.3 KiB/s wr, 2 op/s 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2853160790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1467881906' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1323487149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1515834972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi102-75928-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:57.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: pgmap v102: 520 pgs: 456 active+clean, 64 unknown; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.3 KiB/s wr, 2 op/s 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2853160790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1467881906' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1323487149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1515834972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.126 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi102-75928-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:57.127 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.214 INFO:tasks.workunit.client.0.smithi102.stdout: a api_watch_notify: Running main() from gmock_main.cc 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify_test_cb 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (340 ms) 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855656995120 err -107 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-30T13:59:57.215 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (20 ms) 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855657006832 err -107 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1009 ms) 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 335007449088 cookie 93855657010304 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (6 ms) 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 335007449088 cookie 93855657010304 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (7 ms) 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-30T13:59:57.216 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 335007449090 cookie 93855657010304 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (6 ms) 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 335007449088 cookie 93855657040032 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 335007449088 cookie 93855657048480 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (9 ms) 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 335007449089 cookie 93855657048480 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 335007449090 cookie 93855657048480 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855657048480 err -107 2024-03-30T13:59:57.217 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_cb from 24646 notify_id 365072220162 cookie 93855657040032 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93855657040032 err -107 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1003 ms) 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10413 ms total) 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: watch_notify_test_cb 2024-03-30T13:59:57.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (2828 ms) 2024-03-30T13:59:57.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (2828 ms total) 2024-03-30T13:59:57.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: 2024-03-30T13:59:57.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-30T13:59:57.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (33808 ms total) 2024-03-30T13:59:57.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-30T13:59:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: pgmap v102: 520 pgs: 456 active+clean, 64 unknown; 145 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.3 KiB/s wr, 2 op/s 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2853160790' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/622777449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1467881906' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1323487149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1515834972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]: dispatch 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi102-75928-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T13:59:57.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:58.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: pgmap v105: 720 pgs: 100 creating+peering, 509 active+clean, 111 unknown; 145 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-30T13:59:58.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T13:59:58.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: pgmap v105: 720 pgs: 100 creating+peering, 509 active+clean, 111 unknown; 145 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.441 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T13:59:58.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:58.442 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:58 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T13:59:58.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: pgmap v105: 720 pgs: 100 creating+peering, 509 active+clean, 111 unknown; 145 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi102-75270-12"}]': finished 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi102-74805-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T13:59:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:58 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:59 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:00.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:59 smithi138 ceph-mon[33397]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T14:00:00.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/647131572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 13:59:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[29392]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/647131572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[37717]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T14:00:00.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/647131572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:00.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 13:59:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:00.894 INFO:tasks.workunit.client.0.smithi102.stdout:fault" 2024-03-30T14:00:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: }, 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: "task_status": {} 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: } 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: } 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: } 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: } 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: } 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: attempt 1 of 20 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (11490 ms) 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (37425 ms total) 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-30T14:00:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (37425 ms total) 2024-03-30T14:00:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-30T14:00:01.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: pgmap v107: 624 pgs: 94 creating+peering, 419 active+clean, 111 unknown; 145 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 7 pool(s) do not have an application enabled 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: pool 'test-rados-api-smithi102-75421-6' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: [WRN] POOL_APP_NOT_ENABLED: 7 pool(s) do not have an application enabled 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: application not enabled on pool 'SnapGetNamesmithi102-76097-4' 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: application not enabled on pool 'WriteSamesmithi102-76139-6' 2024-03-30T14:00:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: application not enabled on pool 'ListObjectsManysmithi102-75928-3' 2024-03-30T14:00:01.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: application not enabled on pool 'ReadIntoBuufferlistsmithi102-75892-6' 2024-03-30T14:00:01.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: application not enabled on pool 'Execsmithi102-76052-7' 2024-03-30T14:00:01.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:01 smithi138 ceph-mon[33397]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: pgmap v107: 624 pgs: 94 creating+peering, 419 active+clean, 111 unknown; 145 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 7 pool(s) do not have an application enabled 2024-03-30T14:00:01.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: pool 'test-rados-api-smithi102-75421-6' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: [WRN] POOL_APP_NOT_ENABLED: 7 pool(s) do not have an application enabled 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: application not enabled on pool 'SnapGetNamesmithi102-76097-4' 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: application not enabled on pool 'WriteSamesmithi102-76139-6' 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: application not enabled on pool 'ListObjectsManysmithi102-75928-3' 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: application not enabled on pool 'ReadIntoBuufferlistsmithi102-75892-6' 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: application not enabled on pool 'Execsmithi102-76052-7' 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[29392]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: pgmap v107: 624 pgs: 94 creating+peering, 419 active+clean, 111 unknown; 145 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-03-30T14:00:01.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi102-75095-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 7 pool(s) do not have an application enabled 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: pool 'test-rados-api-smithi102-75421-6' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: [WRN] POOL_APP_NOT_ENABLED: 7 pool(s) do not have an application enabled 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: application not enabled on pool 'SnapGetNamesmithi102-76097-4' 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: application not enabled on pool 'WriteSamesmithi102-76139-6' 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: application not enabled on pool 'ListObjectsManysmithi102-75928-3' 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: application not enabled on pool 'ReadIntoBuufferlistsmithi102-75892-6' 2024-03-30T14:00:01.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: application not enabled on pool 'Execsmithi102-76052-7' 2024-03-30T14:00:01.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:01 smithi102 ceph-mon[37717]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: pgmap v110: 560 pgs: 107 creating+peering, 428 active+clean, 25 unknown; 145 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 1 op/s 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/95671662' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2253316021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3444234167' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2653992732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T14:00:02.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:02.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: pgmap v110: 560 pgs: 107 creating+peering, 428 active+clean, 25 unknown; 145 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 1 op/s 2024-03-30T14:00:02.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/95671662' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2253316021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3444234167' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2653992732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:02.690 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: pgmap v110: 560 pgs: 107 creating+peering, 428 active+clean, 25 unknown; 145 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 1 op/s 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/95671662' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2253316021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3444234167' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.691 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2653992732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:02.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:02.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:00:02.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T14:00:02.693 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:03.347 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: Running main() from gmock_main.cc 2024-03-30T14:00:03.347 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-30T14:00:03.347 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [----------] Global test environment set-up. 2024-03-30T14:00:03.347 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-30T14:00:03.347 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-30T14:00:03.347 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (18728 ms) 2024-03-30T14:00:03.347 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (10088 ms) 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (10912 ms) 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [----------] 3 tests from NeoradosECList (39728 ms total) 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [----------] Global test environment tear-down 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (39728 ms total) 2024-03-30T14:00:03.348 INFO:tasks.workunit.client.0.smithi102.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-30T14:00:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: pgmap v113: 680 pgs: 117 creating+peering, 435 active+clean, 128 unknown; 145 MiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1503600356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:04.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: pgmap v113: 680 pgs: 117 creating+peering, 435 active+clean, 128 unknown; 145 MiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1503600356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: pgmap v113: 680 pgs: 117 creating+peering, 435 active+clean, 128 unknown; 145 MiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75462-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi102-74805-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi102-75928-3"}]: dispatch 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1503600356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:04.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:05.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]': finished 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:05.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3981201272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1996716756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]': finished 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:00:05.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3981201272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1996716756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.191 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1707634336' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi102-75928-3"}]': finished 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-7"}]': finished 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:00:05.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:05.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]: dispatch 2024-03-30T14:00:05.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3981201272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1996716756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:05.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:00:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:00:06.629 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: Running main() from gmock_main.cc 2024-03-30T14:00:06.629 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-30T14:00:06.629 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [----------] Global test environment set-up. 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (11075 ms) 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (6814 ms) 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3747 ms) 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (4030 ms) 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (4102 ms) 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-30T14:00:06.630 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (5060 ms) 2024-03-30T14:00:06.631 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-30T14:00:06.631 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (8105 ms) 2024-03-30T14:00:06.631 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (42933 ms total) 2024-03-30T14:00:06.631 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: 2024-03-30T14:00:06.631 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [----------] Global test environment tear-down 2024-03-30T14:00:06.631 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (42933 ms total) 2024-03-30T14:00:06.631 INFO:tasks.workunit.client.0.smithi102.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-30T14:00:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: pgmap v116: 648 pgs: 53 creating+peering, 435 active+clean, 160 unknown; 145 MiB data, 791 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 0 op/s 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]: dispatch 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]: dispatch 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]': finished 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]': finished 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T14:00:06.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: pgmap v116: 648 pgs: 53 creating+peering, 435 active+clean, 160 unknown; 145 MiB data, 791 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 0 op/s 2024-03-30T14:00:06.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]: dispatch 2024-03-30T14:00:06.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]: dispatch 2024-03-30T14:00:06.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]': finished 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]': finished 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T14:00:06.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:00:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:06.948 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: pgmap v116: 648 pgs: 53 creating+peering, 435 active+clean, 160 unknown; 145 MiB data, 791 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 0 op/s 2024-03-30T14:00:06.948 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]: dispatch 2024-03-30T14:00:06.948 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]: dispatch 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-7"}]': finished 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi102-74825-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi102-74805-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4"}]': finished 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T14:00:06.949 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:00:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]': finished 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:00:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:00:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:00:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:00:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:00:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]': finished 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:00:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:00:08.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:00:08.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:00:08.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:08.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:08.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi102-75421-4", "tierpool": "test-rados-api-smithi102-75421-6"}]': finished 2024-03-30T14:00:08.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:00:08.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1876178579' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:00:08.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:00:08.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: 2024-03-30T14:00:08.819 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (12258 ms) 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (6638 ms) 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3507 ms) 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3509 ms) 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4084 ms) 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (5087 ms) 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-30T14:00:08.820 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (10276 ms) 2024-03-30T14:00:08.821 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (45359 ms total) 2024-03-30T14:00:08.821 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: 2024-03-30T14:00:08.821 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-30T14:00:08.821 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (45359 ms total) 2024-03-30T14:00:08.821 INFO:tasks.workunit.client.0.smithi102.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-30T14:00:08.836 INFO:tasks.workunit.client.0.smithi102.stdout:pi_watch_notify_pp: flushed 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3016 ms) 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: trying... 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241098224 notify_id 416611827713 notifier_gid 25186 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: timed out 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: flushing 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: flushed 2024-03-30T14:00:08.837 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3021 ms) 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: List watches 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: notify2 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241098224 notify_id 429496729602 notifier_gid 25186 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: notify2 done 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: watch_check 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: unwatch2 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: flushing 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: done 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3020 ms) 2024-03-30T14:00:08.838 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: List watches 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: notify2 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: handle_notify cookie 94846241114304 notify_id 442381631490 notifier_gid 25186 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: notify2 done 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: watch_check 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: unwatch2 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: flushing 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: done 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3014 ms) 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18053 ms total) 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (45373 ms total) 2024-03-30T14:00:08.839 INFO:tasks.workunit.client.0.smithi102.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: pgmap v119: 648 pgs: 64 creating+peering, 584 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:00:09.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.0 deep-scrub starts 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.0 deep-scrub ok 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.3 deep-scrub starts 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.3 deep-scrub ok 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.4 deep-scrub starts 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.4 deep-scrub ok 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.6 deep-scrub starts 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.6 deep-scrub ok 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.7 deep-scrub starts 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.7 deep-scrub ok 2024-03-30T14:00:09.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.1 deep-scrub starts 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: 11.1 deep-scrub ok 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1069378298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2482481451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: pgmap v119: 648 pgs: 64 creating+peering, 584 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-30T14:00:09.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:00:09.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:00:09.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:00:09.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:00:09.190 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.0 deep-scrub starts 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.0 deep-scrub ok 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.3 deep-scrub starts 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.3 deep-scrub ok 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.4 deep-scrub starts 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.4 deep-scrub ok 2024-03-30T14:00:09.191 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.6 deep-scrub starts 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.6 deep-scrub ok 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.7 deep-scrub starts 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.7 deep-scrub ok 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.1 deep-scrub starts 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: 11.1 deep-scrub ok 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1069378298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2482481451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:00:09.192 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.193 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: pgmap v119: 648 pgs: 64 creating+peering, 584 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-03-30T14:00:09.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:00:09.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:00:09.193 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.0 deep-scrub starts 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.0 deep-scrub ok 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.3 deep-scrub starts 2024-03-30T14:00:09.194 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.3 deep-scrub ok 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.4 deep-scrub starts 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.4 deep-scrub ok 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.6 deep-scrub starts 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.6 deep-scrub ok 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.7 deep-scrub starts 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.7 deep-scrub ok 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.1 deep-scrub starts 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: 11.1 deep-scrub ok 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi102-75421-6", "pool2": "test-rados-api-smithi102-75421-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1069378298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2482481451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.195 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:09.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:00:09.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:09.196 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: 11.5 deep-scrub starts 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: 11.5 deep-scrub ok 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: pgmap v122: 612 pgs: 160 unknown, 452 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 op/s 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3577530717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:10.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:10.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: 11.5 deep-scrub starts 2024-03-30T14:00:10.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: 11.5 deep-scrub ok 2024-03-30T14:00:10.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: pgmap v122: 612 pgs: 160 unknown, 452 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 op/s 2024-03-30T14:00:10.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3577530717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: 11.5 deep-scrub starts 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: 11.5 deep-scrub ok 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: pgmap v122: 612 pgs: 160 unknown, 452 active+clean; 145 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 op/s 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi102-74805-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi102-74825-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3577530717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:10.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:10.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:10.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] Global test environment set-up. 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: 2024-03-30T13:59:23.409+0000 7f70fe6d9780 -1 failed for service _ceph-mon._tcp 2024-03-30T14:00:11.085 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: 2024-03-30T13:59:23.409+0000 7f70fe6d9780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-30T14:00:11.086 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (64 ms) 2024-03-30T14:00:11.086 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-30T14:00:11.086 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2009 ms) 2024-03-30T14:00:11.086 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2073 ms total) 2024-03-30T14:00:11.086 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: 2024-03-30T14:00:11.086 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-30T14:00:11.086 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-30T14:00:11.087 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: started 0x5583ecd5c660 2024-03-30T14:00:11.087 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: started 0x5583ed091ca0 2024-03-30T14:00:11.087 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: started 2 aios 2024-03-30T14:00:11.087 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: waiting 0x5583ecd5c660 2024-03-30T14:00:11.087 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: waiting 0x5583ed091ca0 2024-03-30T14:00:11.087 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: done. 2024-03-30T14:00:11.087 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (15078 ms) 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (15078 ms total) 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.Exec (106 ms) 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (33 ms) 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (5 ms) 2024-03-30T14:00:11.088 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5458 ms) 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22692 ms) 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28294 ms total) 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [----------] Global test environment tear-down 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (47731 ms total) 2024-03-30T14:00:11.089 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3577530717' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: pgmap v125: 516 pgs: 64 creating+peering, 452 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]': finished 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]: dispatch 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/380074633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4279481307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]: dispatch 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3577530717' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:11.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:11.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:11.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:11.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: pgmap v125: 516 pgs: 64 creating+peering, 452 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]': finished 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/380074633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4279481307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3577530717' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: pgmap v125: 516 pgs: 64 creating+peering, 452 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-9"}]': finished 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:11.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T14:00:11.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/380074633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4279481307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]: dispatch 2024-03-30T14:00:11.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:11.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:12.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]': finished 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: 11.2 deep-scrub starts 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: 11.2 deep-scrub ok 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]': finished 2024-03-30T14:00:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: 11.2 deep-scrub starts 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: 11.2 deep-scrub ok 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:12.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi102-75092-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-9", "mode": "writeback"}]': finished 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi102-74805-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi102-74825-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: 11.2 deep-scrub starts 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: 11.2 deep-scrub ok 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:12.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:13.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:13.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:13 smithi138 ceph-mon[33397]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:00:13.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:13.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:13.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[29392]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:00:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:13.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:13.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:13.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[37717]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:00:13.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:13.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]: dispatch 2024-03-30T14:00:13.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: Running main() from gmock_main.cc 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [----------] Global test environment set-up. 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.Version (9546 ms) 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4899 ms) 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.LongName (4539 ms) 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-30T14:00:14.644 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3200 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3504 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4114 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.Exec (5001 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3892 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3275 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3193 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2246 ms) 2024-03-30T14:00:14.645 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-30T14:00:14.646 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3568 ms) 2024-03-30T14:00:14.646 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [----------] 12 tests from NeoRadosMisc (50977 ms total) 2024-03-30T14:00:14.646 INFO:tasks.workunit.client.0.smithi102.stdout: misc: 2024-03-30T14:00:14.646 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [----------] Global test environment tear-down 2024-03-30T14:00:14.646 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [==========] 12 tests from 1 test suite ran. (50978 ms total) 2024-03-30T14:00:14.646 INFO:tasks.workunit.client.0.smithi102.stdout: misc: [ PASSED ] 12 tests. 2024-03-30T14:00:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: pgmap v128: 524 pgs: 104 unknown, 420 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: 11.9 deep-scrub starts 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: 11.9 deep-scrub ok 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]': finished 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/624556169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi102-74825-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/494168330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: 11.8 deep-scrub starts 2024-03-30T14:00:14.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: 11.8 deep-scrub ok 2024-03-30T14:00:14.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: pgmap v128: 524 pgs: 104 unknown, 420 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:00:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: 11.9 deep-scrub starts 2024-03-30T14:00:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: 11.9 deep-scrub ok 2024-03-30T14:00:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]': finished 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/624556169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi102-74825-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/494168330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: 11.8 deep-scrub starts 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: 11.8 deep-scrub ok 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: pgmap v128: 524 pgs: 104 unknown, 420 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:00:14.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: 11.9 deep-scrub starts 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: 11.9 deep-scrub ok 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-9"}]': finished 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/624556169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi102-74825-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/494168330' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: 11.8 deep-scrub starts 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: 11.8 deep-scrub ok 2024-03-30T14:00:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:15.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:00:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:00:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:15 smithi138 ceph-mon[33397]: pgmap v131: 556 pgs: 200 unknown, 356 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/624556169' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi102-74825-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:15 smithi138 ceph-mon[33397]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:00:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:15.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[29392]: pgmap v131: 556 pgs: 200 unknown, 356 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/624556169' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi102-74825-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[29392]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[37717]: pgmap v131: 556 pgs: 200 unknown, 356 active+clean; 217 MiB data, 1012 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/624556169' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi102-74825-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi102-74805-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75103-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[37717]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:00:15.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:16.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:16.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:00:16.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:00:16.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:16.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:00:16.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:16.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:16.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:16 smithi138 ceph-mon[33397]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:16.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:00:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:16.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:16.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:00:16.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:00:16.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:16.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[37717]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:16.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:16 smithi102 ceph-mon[29392]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: pgmap v134: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]': finished 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2986800635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2697925327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi102-74805-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:17.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: pgmap v134: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-03-30T14:00:17.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]': finished 2024-03-30T14:00:17.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:17.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:00:17.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2986800635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2697925327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi102-74805-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: pgmap v134: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache", "force_nonempty":""}]': finished 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2986800635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2697925327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi102-74805-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:17.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2697925327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi102-74805-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]: dispatch 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]: dispatch 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:19.099 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]': finished 2024-03-30T14:00:19.100 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:19.100 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:19.100 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T14:00:19.100 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:19.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2697925327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi102-74805-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]: dispatch 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]: dispatch 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]': finished 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T14:00:19.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75103-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2697925327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi102-74805-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2579540407' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]: dispatch 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]: dispatch 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75103-10", "tierpool":"test-rados-api-smithi102-75103-10-cache"}]': finished 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:19.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T14:00:19.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:20.049 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: pgmap v137: 524 pgs: 128 unknown, 396 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2024-03-30T14:00:20.049 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:20.049 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]': finished 2024-03-30T14:00:20.049 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]: dispatch 2024-03-30T14:00:20.049 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T14:00:20.049 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3818676763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/559369219' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]: dispatch 2024-03-30T14:00:20.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: pgmap v137: 524 pgs: 128 unknown, 396 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2024-03-30T14:00:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]': finished 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3818676763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/559369219' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: pgmap v137: 524 pgs: 128 unknown, 396 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-11"}]': finished 2024-03-30T14:00:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]: dispatch 2024-03-30T14:00:20.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T14:00:20.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3818676763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/559369219' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]: dispatch 2024-03-30T14:00:20.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:20.664 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: Running main() from gmock_main.cc 2024-03-30T14:00:20.664 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-30T14:00:20.664 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [----------] Global test environment set-up. 2024-03-30T14:00:20.664 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-30T14:00:20.664 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-30T14:00:20.664 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolList (11403 ms) 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4696 ms) 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (4740 ms) 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3776 ms) 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4470 ms) 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (7740 ms) 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6443 ms) 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-30T14:00:20.665 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5924 ms) 2024-03-30T14:00:20.666 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-30T14:00:20.666 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8115 ms) 2024-03-30T14:00:20.666 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [----------] 9 tests from LibRadosPools (57307 ms total) 2024-03-30T14:00:20.666 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: 2024-03-30T14:00:20.666 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [----------] Global test environment tear-down 2024-03-30T14:00:20.666 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (57307 ms total) 2024-03-30T14:00:20.666 INFO:tasks.workunit.client.0.smithi102.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-30T14:00:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:21.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:20 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:21.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]': finished 2024-03-30T14:00:21.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:21.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:21.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:20 smithi138 ceph-mon[33397]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T14:00:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]': finished 2024-03-30T14:00:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[29392]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T14:00:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-11", "mode": "writeback"}]': finished 2024-03-30T14:00:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi102-74805-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi102-74825-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:20 smithi102 ceph-mon[37717]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T14:00:22.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:22 smithi138 ceph-mon[33397]: pgmap v140: 524 pgs: 96 creating+peering, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 258 KiB/s wr, 5 op/s 2024-03-30T14:00:22.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:22.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:22 smithi138 ceph-mon[33397]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:22.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:22 smithi138 ceph-mon[33397]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:00:22.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:22.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:22.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:22.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[29392]: pgmap v140: 524 pgs: 96 creating+peering, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 258 KiB/s wr, 5 op/s 2024-03-30T14:00:22.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:22.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[29392]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:22.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[29392]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:00:22.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:22.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[37717]: pgmap v140: 524 pgs: 96 creating+peering, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 258 KiB/s wr, 5 op/s 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[37717]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[37717]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:23.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1312758910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/471562398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:23.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:23.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:23.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1312758910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/471562398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1765269273' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1312758910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/471562398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]: dispatch 2024-03-30T14:00:23.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:23.440 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: Running main() from gmock_main.cc 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: [----------] Global test environment set-up. 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.Limits (12117 ms) 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-30T14:00:23.552 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (5843 ms) 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.ReadOp (3469 ms) 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.SparseRead (3926 ms) 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.RoundTrip (4477 ms) 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (5030 ms) 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3884 ms) 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3270 ms) 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-30T14:00:23.553 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3196 ms) 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.Trunc (2719 ms) 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.Remove (3093 ms) 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3027 ms) 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.RmXattr (3000 ms) 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2885 ms) 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [----------] 14 tests from NeoRadosIo (59936 ms total) 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [----------] Global test environment tear-down 2024-03-30T14:00:23.554 INFO:tasks.workunit.client.0.smithi102.stdout: io: [==========] 14 tests from 1 test suite ran. (59936 ms total) 2024-03-30T14:00:23.555 INFO:tasks.workunit.client.0.smithi102.stdout: io: [ PASSED ] 14 tests. 2024-03-30T14:00:24.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: pgmap v143: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 2 op/s 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]': finished 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:24.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:24.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: pgmap v143: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 2 op/s 2024-03-30T14:00:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]': finished 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: pgmap v143: 420 pgs: 32 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 2 op/s 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi102-75092-10"}]': finished 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi102-74825-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-11"}]': finished 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi102-74805-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:24.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:25.614 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[29392]: pgmap v146: 452 pgs: 64 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:25.614 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:25.614 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[29392]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:00:25.614 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:25.614 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:25.614 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:25.615 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[37717]: pgmap v146: 452 pgs: 64 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:25.615 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:25.615 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[37717]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:00:25.615 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:25.615 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:25.615 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:25.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:00:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:00:25.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:25 smithi138 ceph-mon[33397]: pgmap v146: 452 pgs: 64 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:25.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:25.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:25 smithi138 ceph-mon[33397]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:00:25.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:25.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:25.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:26.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:26.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3821256545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/731502543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:26 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:26.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:00:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:26.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:26.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:00:26.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3821256545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/731502543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3821256545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/731502543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:26.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:26.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:26.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:26 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:27 smithi138 ceph-mon[33397]: pgmap v149: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-03-30T14:00:27.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:27.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:27 smithi138 ceph-mon[33397]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:00:27.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[29392]: pgmap v149: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[29392]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[37717]: pgmap v149: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi102-75092-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:27.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi102-74805-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi102-74825-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[37717]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:00:27.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:28.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:28 smithi138 ceph-mon[33397]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T14:00:28.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:28.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:28 smithi138 ceph-mon[33397]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T14:00:28.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2991981035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1672162095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi102-74805-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[37717]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T14:00:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[37717]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T14:00:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2991981035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1672162095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi102-74805-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[29392]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T14:00:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[29392]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T14:00:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2991981035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1672162095' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi102-74805-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:29 smithi138 ceph-mon[33397]: pgmap v152: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T14:00:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1672162095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi102-74805-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:29 smithi138 ceph-mon[33397]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T14:00:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[29392]: pgmap v152: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T14:00:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1672162095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi102-74805-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[29392]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T14:00:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[37717]: pgmap v152: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T14:00:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1672162095' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi102-74805-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi102-74825-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:29 smithi102 ceph-mon[37717]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T14:00:30.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:30.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:30 smithi138 ceph-mon[33397]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:00:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:30 smithi102 ceph-mon[29392]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:00:30.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:30.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:30 smithi102 ceph-mon[37717]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: pgmap v155: 460 pgs: 460 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 357 KiB/s wr, 98 op/s 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]': finished 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/308198071' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3273036879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:31.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: pgmap v155: 460 pgs: 460 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 357 KiB/s wr, 98 op/s 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]': finished 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/308198071' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3273036879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: pgmap v155: 460 pgs: 460 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 357 KiB/s wr, 98 op/s 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:31.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]': finished 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/308198071' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3273036879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:31.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: pgmap v158: 460 pgs: 64 unknown, 396 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 360 KiB/s wr, 102 op/s 2024-03-30T14:00:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]': finished 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]: dispatch 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]: dispatch 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: pgmap v158: 460 pgs: 64 unknown, 396 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 360 KiB/s wr, 102 op/s 2024-03-30T14:00:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]': finished 2024-03-30T14:00:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T14:00:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:00:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]: dispatch 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]: dispatch 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: pgmap v158: 460 pgs: 64 unknown, 396 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 360 KiB/s wr, 102 op/s 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-13"}]': finished 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi102-74805-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi102-74825-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T14:00:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:00:33.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]: dispatch 2024-03-30T14:00:33.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:33.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]: dispatch 2024-03-30T14:00:33.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:33.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:33.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T14:00:34.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]': finished 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]': finished 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-30T14:00:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-30T14:00:34.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-30T14:00:34.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-30T14:00:34.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-30T14:00:34.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-30T14:00:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]': finished 2024-03-30T14:00:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]': finished 2024-03-30T14:00:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:00:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:00:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-13", "mode": "writeback"}]': finished 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pg_num","val":"11"}]': finished 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-30T14:00:34.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-30T14:00:35.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:00:35.572 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-30T14:00:35.572 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-30T14:00:35.573 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-30T14:00:35.573 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-30T14:00:35.573 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-30T14:00:35.573 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-30T14:00:35.573 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-30T14:00:35.573 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-30T14:00:35.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: pgmap v161: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.2 scrub starts 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.2 scrub ok 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1426906593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3827339202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.6 scrub starts 2024-03-30T14:00:35.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.6 scrub ok 2024-03-30T14:00:35.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.7 scrub starts 2024-03-30T14:00:35.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.7 scrub ok 2024-03-30T14:00:35.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.0 scrub starts 2024-03-30T14:00:35.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: 161.0 scrub ok 2024-03-30T14:00:35.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:35.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: pgmap v161: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.2 scrub starts 2024-03-30T14:00:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.2 scrub ok 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1426906593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3827339202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.6 scrub starts 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.6 scrub ok 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.7 scrub starts 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.7 scrub ok 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.0 scrub starts 2024-03-30T14:00:35.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: 161.0 scrub ok 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-30T14:00:35.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: pgmap v161: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.2 scrub starts 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.2 scrub ok 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1426906593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3827339202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.6 scrub starts 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.6 scrub ok 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.7 scrub starts 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.7 scrub ok 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.0 scrub starts 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: 161.0 scrub ok 2024-03-30T14:00:35.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.1 scrub starts 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.1 scrub ok 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.9 scrub starts 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.9 scrub ok 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.5 scrub starts 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.5 scrub ok 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.8 scrub starts 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.8 scrub ok 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.3 scrub starts 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: 161.3 scrub ok 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:36.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:36.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:00:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:36.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.1 scrub starts 2024-03-30T14:00:36.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.1 scrub ok 2024-03-30T14:00:36.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.9 scrub starts 2024-03-30T14:00:36.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.9 scrub ok 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.5 scrub starts 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.5 scrub ok 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.8 scrub starts 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.8 scrub ok 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.3 scrub starts 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: 161.3 scrub ok 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:00:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.1 scrub starts 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.1 scrub ok 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.9 scrub starts 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.9 scrub ok 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.5 scrub starts 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.5 scrub ok 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.8 scrub starts 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.8 scrub ok 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi102-74805-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:36.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi102-74825-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.3 scrub starts 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: 161.3 scrub ok 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:36.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (4744 ms) 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-30T14:00:37.593 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (1622 ms) 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2155 ms) 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2669 ms) 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (11190 ms total) 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (6105 ms) 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (6352 ms) 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-30T14:00:37.594 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5301 ms) 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (17758 ms total) 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3097 ms) 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2025 ms) 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2000 ms) 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1999 ms) 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9121 ms total) 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: 2024-03-30T14:00:37.595 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4006 ms) 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4014 ms) 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8020 ms total) 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (74252 ms total) 2024-03-30T14:00:37.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-30T14:00:37.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: pgmap v164: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 108 op/s 2024-03-30T14:00:37.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: 161.4 deep-scrub starts 2024-03-30T14:00:37.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: 161.4 deep-scrub ok 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2"}]: dispatch 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:37.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: pgmap v164: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 108 op/s 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: 161.4 deep-scrub starts 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: 161.4 deep-scrub ok 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T14:00:37.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2"}]: dispatch 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: pgmap v164: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 108 op/s 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: 161.4 deep-scrub starts 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: 161.4 deep-scrub ok 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:37.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T14:00:37.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1289845223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:37.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]: dispatch 2024-03-30T14:00:37.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2"}]: dispatch 2024-03-30T14:00:37.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:37.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:37.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:38.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:38.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:38.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2699893560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3700110370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2699893560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3700110370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi102-75092-15"}]': finished 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi102-74944-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2699893560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3700110370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:38.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:00:38.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: pgmap v167: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 25 KiB/s wr, 108 op/s 2024-03-30T14:00:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:00:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:00:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:00:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:00:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: pgmap v167: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 25 KiB/s wr, 108 op/s 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:00:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: pgmap v167: 452 pgs: 96 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 25 KiB/s wr, 108 op/s 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi102-74825-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi102-74805-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:00:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:00:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:40 smithi138 ceph-mon[33397]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:00:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:00:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:00:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:40 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-30T14:00:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:40 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-30T14:00:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:00:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[37717]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:00:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:00:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:00:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-30T14:00:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[29392]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1281407755' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-30T14:00:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:41.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: seed 74944 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (362 ms) 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-30T14:00:41.619 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (13 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (10 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (9 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (6 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (7 ms) 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-30T14:00:41.620 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4 ms) 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (7 ms) 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1477 ms) 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (371 ms) 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: done waiting, doing copies 2024-03-30T14:00:41.621 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: done waiting 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60933 ms) 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (5 ms) 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4186 ms) 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-30T14:00:41.622 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-30T14:00:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: pgmap v170: 388 pgs: 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3301405804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2283832916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: pgmap v170: 388 pgs: 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:00:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:00:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-30T14:00:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:00:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3301405804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2283832916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: pgmap v170: 388 pgs: 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi102-74944-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi102-74889-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:00:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3301405804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2283832916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:41.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T14:00:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3301405804' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T14:00:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:00:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3301405804' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T14:00:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:00:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3301405804' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74805-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi102-74889-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992304861' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:42.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:42.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: pgmap v173: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:00:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2508154004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75095-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi102-74889-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: pgmap v173: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2508154004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75095-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi102-74889-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: pgmap v173: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-74944-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2508154004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75095-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi102-74889-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2508154004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75095-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi102-74889-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:44 smithi138 ceph-mon[33397]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T14:00:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi102-74889-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:44.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2724984538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3347393454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi102-74805-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2508154004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75095-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi102-74889-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[37717]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi102-74889-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2724984538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3347393454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi102-74805-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2508154004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75095-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:44.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi102-74889-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:44.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[29392]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T14:00:44.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi102-74889-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:44.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2724984538' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3347393454' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi102-74805-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:44.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:45.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:00:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:00:45.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:45 smithi138 ceph-mon[33397]: pgmap v176: 388 pgs: 96 unknown, 292 active+clean; 590 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:45 smithi138 ceph-mon[33397]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3347393454' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi102-74805-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:45 smithi138 ceph-mon[33397]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T14:00:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[29392]: pgmap v176: 388 pgs: 96 unknown, 292 active+clean; 590 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[29392]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3347393454' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi102-74805-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[29392]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T14:00:45.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[37717]: pgmap v176: 388 pgs: 96 unknown, 292 active+clean; 590 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:00:45.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[37717]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:45.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-74944-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:45.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3347393454' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi102-74805-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:45.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi102-74825-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:45.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:45 smithi102 ceph-mon[37717]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T14:00:46.735 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (ry=12 expected=12 2024-03-30T14:00:46.735 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-03-30T14:00:46.735 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=1 expected=1 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=10 expected=10 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=5 expected=5 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-03-30T14:00:46.736 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=11 expected=11 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=0 expected=0 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=14 expected=14 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=6 expected=6 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2024-03-30T14:00:46.737 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: : entry=8 expected=8 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (185 ms) 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (58683 ms) 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (9080 ms) 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] 7 tests from LibRadosList (68505 ms total) 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-30T14:00:46.738 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1046 ms) 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset foo1,foo2,foo3 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo1 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo2 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo3 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset foo1,foo4,foo5 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo4 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo5 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo1 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset foo6,foo7 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo7 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: foo6 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-30T14:00:46.739 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns1:foo4 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns1:foo5 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns2:foo7 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns2:foo6 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: ns1:foo1 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: :foo1 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: :foo2 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: :foo3 2024-03-30T14:00:46.740 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (29 ms) 2024-03-30T14:00:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi102-74889-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:46 smithi138 ceph-mon[33397]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T14:00:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:46.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:00:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi102-74889-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[37717]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T14:00:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:46.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi102-74889-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[29392]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T14:00:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: pgmap v179: 372 pgs: 8 creating+peering, 40 unknown, 324 active+clean; 590 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:47.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:47.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:00:47.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1995146213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/310221328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:47.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: pgmap v179: 372 pgs: 8 creating+peering, 40 unknown, 324 active+clean; 590 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:00:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1995146213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/310221328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: pgmap v179: 372 pgs: 8 creating+peering, 40 unknown, 324 active+clean; 590 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1995146213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/310221328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:47.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:00:49.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:49.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:00:49.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi102-75095-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:49.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:49.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:00:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi102-75095-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74944-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi102-74805-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi102-74825-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi102-75095-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:50.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: pgmap v182: 396 pgs: 8 creating+peering, 96 unknown, 292 active+clean; 590 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:50.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:00:50.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:50.401 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi102-75095-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:00:50.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:50.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:00:50.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1063156928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:50.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:50.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/193908575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi102-74805-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: pgmap v182: 396 pgs: 8 creating+peering, 96 unknown, 292 active+clean; 590 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:50.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi102-75095-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1063156928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/193908575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi102-74805-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: pgmap v182: 396 pgs: 8 creating+peering, 96 unknown, 292 active+clean; 590 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]: dispatch 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi102-75095-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2346055619' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi102-74889-2"}]': finished 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1063156928' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:50.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/193908575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi102-74805-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 1 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 10 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 13 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 7 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 14 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 0 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 15 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 11 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 5 0 2024-03-30T14:00:52.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 8 0 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 6 0 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 3 0 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 4 0 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 12 0 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 9 0 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 2 0 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (41 ms) 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] 3 tests from LibRadosListEC (1117 ms total) 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-30T14:00:52.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2873 ms) 2024-03-30T14:00:52.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] 1 test from LibRadosListNP (2873 ms total) 2024-03-30T14:00:52.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: 2024-03-30T14:00:52.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [----------] Global test environment tear-down 2024-03-30T14:00:52.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [==========] 11 tests from 3 test suites ran. (89252 ms total) 2024-03-30T14:00:52.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_list: [ PASSED ] 11 tests. 2024-03-30T14:00:52.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: pgmap v185: 364 pgs: 72 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:52.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:52.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/193908575' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi102-74805-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3779156299' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:52 smithi138 ceph-mon[33397]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: pgmap v185: 364 pgs: 72 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/193908575' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi102-74805-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3779156299' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[37717]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: pgmap v185: 364 pgs: 72 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi102-74825-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/193908575' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi102-74805-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3313214066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3779156299' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]: dispatch 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-74889-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-74944-24"}]': finished 2024-03-30T14:00:52.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:52 smithi102 ceph-mon[29392]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:00:53.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3779156299' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T14:00:53.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: pgmap v188: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:53.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T14:00:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-30T14:00:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:00:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1051041924' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi102-74944-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1193103963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3403934024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3779156299' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: pgmap v188: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1051041924' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi102-74944-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1193103963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3403934024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3779156299' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: pgmap v188: 332 pgs: 32 unknown, 8 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi102-74889-3","pool2":"test-rados-api-smithi102-74889-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1051041924' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi102-74944-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1193103963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3403934024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:54.558 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-30T14:00:54.558 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: seed 75095 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (5944 ms) 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3495 ms) 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3207 ms) 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (3491 ms) 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (5713 ms) 2024-03-30T14:00:54.559 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (21850 ms total) 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4913 ms) 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (5120 ms) 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6835 ms) 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3999 ms) 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (1916 ms) 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1967-g4e50bbcd/rpm/el9/BUILD/ceph-19.0.0-1967-g4e50bbcd/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-30T14:00:54.560 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi102-75095-7 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18247 ms) 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (41030 ms total) 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (14 ms) 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5077 ms) 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5091 ms total) 2024-03-30T14:00:54.561 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: 2024-03-30T14:00:54.562 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-30T14:00:54.562 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-30T14:00:54.562 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2878 ms) 2024-03-30T14:00:54.562 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-30T14:00:54.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1051041924' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi102-74944-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:54 smithi138 ceph-mon[33397]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1051041924' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi102-74944-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[37717]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1051041924' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi102-74944-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi102-74825-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi102-74805-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:54.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:54 smithi102 ceph-mon[29392]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:00:55.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:55 smithi138 ceph-mon[33397]: pgmap v191: 396 pgs: 96 unknown, 8 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:00:55.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:55 smithi138 ceph-mon[33397]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:00:55.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:00:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:00:55.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:55 smithi102 ceph-mon[29392]: pgmap v191: 396 pgs: 96 unknown, 8 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:00:55.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:55 smithi102 ceph-mon[29392]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:00:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:55 smithi102 ceph-mon[37717]: pgmap v191: 396 pgs: 96 unknown, 8 creating+peering, 292 active+clean; 590 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:00:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:55 smithi102 ceph-mon[37717]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:00:56.583 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnap snapshots: Running main() from gmock_main.cc 2024-03-30T14:00:56.583 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-30T14:00:56.583 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [----------] Global test environment set-up. 2024-03-30T14:00:56.583 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-30T14:00:56.583 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (16825 ms) 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5353 ms) 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (6590 ms) 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (7731 ms) 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7657 ms) 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (44156 ms total) 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-30T14:00:56.584 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4721 ms) 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6115 ms) 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7907 ms) 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5982 ms) 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4039 ms) 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi102-76097-11. 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: Waiting for snaps to purge. 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19991 ms) 2024-03-30T14:00:56.585 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48756 ms total) 2024-03-30T14:00:56.586 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: 2024-03-30T14:00:56.586 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [----------] Global test environment tear-down 2024-03-30T14:00:56.586 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (92912 ms total) 2024-03-30T14:00:56.586 INFO:tasks.workunit.client.0.smithi102.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-30T14:00:56.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:56.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T14:00:56.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1618575937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3540958247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2190327035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi102-74805-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:56.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:56 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:00:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1618575937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3540958247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2190327035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi102-74805-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1618575937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3540958247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2190327035' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi102-74805-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:00:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:56 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:00:57.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:57 smithi138 ceph-mon[33397]: pgmap v194: 396 pgs: 96 unknown, 300 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:00:57.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2190327035' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi102-74805-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:57 smithi138 ceph-mon[33397]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[37717]: pgmap v194: 396 pgs: 96 unknown, 300 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2190327035' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi102-74805-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[37717]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[29392]: pgmap v194: 396 pgs: 96 unknown, 300 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi102-74825-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi102-74944-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2190327035' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi102-74805-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:57.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:57 smithi102 ceph-mon[29392]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T14:00:58.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:58 smithi138 ceph-mon[33397]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T14:00:58.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:58 smithi138 ceph-mon[33397]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T14:00:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1647539524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2061999836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2322534039' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi102-74944-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[29392]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[29392]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1647539524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2061999836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2322534039' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi102-74944-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[37717]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T14:00:58.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[37717]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T14:00:58.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1647539524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2061999836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2322534039' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi102-74944-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:58.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:00:59.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:59 smithi138 ceph-mon[33397]: pgmap v197: 268 pgs: 268 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:00:59.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2322534039' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi102-74944-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:59 smithi138 ceph-mon[33397]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T14:00:59.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:00:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[29392]: pgmap v197: 268 pgs: 268 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:00:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2322534039' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi102-74944-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[29392]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T14:00:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:00:59.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[37717]: pgmap v197: 268 pgs: 268 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:00:59.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2322534039' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi102-74944-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi102-74805-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi102-74825-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:00:59.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[37717]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T14:00:59.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:00:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:01:01.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: pgmap v200: 356 pgs: 356 active+clean; 598 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:01 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:01.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: pgmap v200: 356 pgs: 356 active+clean; 598 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T14:01:01.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:01:01.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T14:01:01.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:01:01.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:01.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: pgmap v200: 356 pgs: 356 active+clean; 598 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:01.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:01.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:01 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:02.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:01:02.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3888578051' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:02.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:02.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:01:02.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:02.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:02.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T14:01:02.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:02.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3888578051' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:02.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1177983802' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi102-75095-16"}]': finished 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi102-74805-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3888578051' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:02.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:02.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:03 smithi138 ceph-mon[33397]: pgmap v203: 292 pgs: 32 unknown, 260 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:03 smithi138 ceph-mon[33397]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T14:01:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[29392]: pgmap v203: 292 pgs: 32 unknown, 260 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[29392]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T14:01:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[37717]: pgmap v203: 292 pgs: 32 unknown, 260 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi102-74825-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[37717]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T14:01:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:05.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:05.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:05.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T14:01:05.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:01:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T14:01:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3194418682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:05.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3194418682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:05.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi102-74944-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:05.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi102-74805-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:05.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T14:01:05.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:01:05.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:05.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T14:01:05.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3194418682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:05.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:05.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:01:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: pgmap v206: 276 pgs: 16 unknown, 260 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:01:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:01:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:06.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: osdmap e167: 8 total, 8 up, 8 in 2024-03-30T14:01:06.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:06.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:06.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:06.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: pgmap v206: 276 pgs: 16 unknown, 260 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:01:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:01:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: osdmap e167: 8 total, 8 up, 8 in 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: pgmap v206: 276 pgs: 16 unknown, 260 active+clean; 590 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi102-74825-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:06.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: osdmap e167: 8 total, 8 up, 8 in 2024-03-30T14:01:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:06.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:06.900 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:01:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: osdmap e168: 8 total, 8 up, 8 in 2024-03-30T14:01:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: osdmap e168: 8 total, 8 up, 8 in 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: osdmap e168: 8 total, 8 up, 8 in 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4273883755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3997105676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]: dispatch 2024-03-30T14:01:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]: dispatch 2024-03-30T14:01:07.609 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: Running main() from gmock_main.cc 2024-03-30T14:01:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-30T14:01:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [----------] Global test environment set-up. 2024-03-30T14:01:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-30T14:01:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-30T14:01:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10317 ms) 2024-03-30T14:01:07.610 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (8801 ms) 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3492 ms) 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3523 ms) 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (4095 ms) 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (5050 ms) 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (4835 ms) 2024-03-30T14:01:07.611 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3261 ms) 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3226 ms) 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2721 ms) 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3085 ms) 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3021 ms) 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-30T14:01:07.612 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2878 ms) 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3005 ms) 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.Flush (3016 ms) 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3014 ms) 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2976 ms) 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3052 ms) 2024-03-30T14:01:07.613 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2995 ms) 2024-03-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3036 ms) 2024-03-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3014 ms) 2024-03-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-30T14:01:07.614 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3267 ms) 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2634 ms) 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3013 ms) 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3011 ms) 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3008 ms) 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [----------] 26 tests from LibRadosAio (97346 ms total) 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-30T14:01:07.615 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout:67404 ms total) 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (104 ms) 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (105 ms total) 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (45 ms) 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-30T14:01:07.616 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (49 ms total) 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (48 ms) 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (5 ms) 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (53 ms total) 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (43 ms) 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-30T14:01:07.617 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (5 ms) 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (48 ms total) 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1042 ms) 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1042 ms total) 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104304 ms total) 2024-03-30T14:01:07.618 INFO:tasks.workunit.client.0.smithi102.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-30T14:01:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: pgmap v209: 300 pgs: 40 creating+peering, 260 active+clean; 589 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: osdmap e169: 8 total, 8 up, 8 in 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4148568700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: pgmap v209: 300 pgs: 40 creating+peering, 260 active+clean; 589 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: osdmap e169: 8 total, 8 up, 8 in 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4148568700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: pgmap v209: 300 pgs: 40 creating+peering, 260 active+clean; 589 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi102-74944-36"}]': finished 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi102-74805-27"}]': finished 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: osdmap e169: 8 total, 8 up, 8 in 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4148568700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:08.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: pgmap v212: 300 pgs: 32 unknown, 8 creating+peering, 260 active+clean; 589 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:01:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: osdmap e170: 8 total, 8 up, 8 in 2024-03-30T14:01:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:01:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: pgmap v212: 300 pgs: 32 unknown, 8 creating+peering, 260 active+clean; 589 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: osdmap e170: 8 total, 8 up, 8 in 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:01:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: pgmap v212: 300 pgs: 32 unknown, 8 creating+peering, 260 active+clean; 589 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi102-74825-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi102-74805-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: osdmap e170: 8 total, 8 up, 8 in 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]: dispatch 2024-03-30T14:01:10.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:10 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]': finished 2024-03-30T14:01:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:10 smithi138 ceph-mon[33397]: osdmap e171: 8 total, 8 up, 8 in 2024-03-30T14:01:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:10 smithi138 ceph-mon[33397]: osdmap e172: 8 total, 8 up, 8 in 2024-03-30T14:01:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2633910306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:11.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]': finished 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[29392]: osdmap e171: 8 total, 8 up, 8 in 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[29392]: osdmap e172: 8 total, 8 up, 8 in 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2633910306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-13"}]': finished 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[37717]: osdmap e171: 8 total, 8 up, 8 in 2024-03-30T14:01:11.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi102-74805-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[37717]: osdmap e172: 8 total, 8 up, 8 in 2024-03-30T14:01:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2633910306' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: pgmap v215: 268 pgs: 268 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-03-30T14:01:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: osdmap e173: 8 total, 8 up, 8 in 2024-03-30T14:01:12.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:12.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: pgmap v215: 268 pgs: 268 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: osdmap e173: 8 total, 8 up, 8 in 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: pgmap v215: 268 pgs: 268 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-03-30T14:01:12.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:12.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:12.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:12.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi102-74825-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:12.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: osdmap e173: 8 total, 8 up, 8 in 2024-03-30T14:01:12.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:12.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:13 smithi138 ceph-mon[33397]: pgmap v218: 308 pgs: 72 unknown, 236 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-30T14:01:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:13 smithi138 ceph-mon[33397]: osdmap e174: 8 total, 8 up, 8 in 2024-03-30T14:01:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[29392]: pgmap v218: 308 pgs: 72 unknown, 236 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[29392]: osdmap e174: 8 total, 8 up, 8 in 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[37717]: pgmap v218: 308 pgs: 72 unknown, 236 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[37717]: osdmap e174: 8 total, 8 up, 8 in 2024-03-30T14:01:13.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:13.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: osdmap e175: 8 total, 8 up, 8 in 2024-03-30T14:01:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/586992336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: osdmap e176: 8 total, 8 up, 8 in 2024-03-30T14:01:14.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: osdmap e175: 8 total, 8 up, 8 in 2024-03-30T14:01:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/586992336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: osdmap e176: 8 total, 8 up, 8 in 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: osdmap e175: 8 total, 8 up, 8 in 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2106691228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:14.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/586992336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]: dispatch 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi102-74805-28"}]': finished 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi102-74825-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: osdmap e176: 8 total, 8 up, 8 in 2024-03-30T14:01:14.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:15.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:01:15.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: pgmap v221: 300 pgs: 64 unknown, 236 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]': finished 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]: dispatch 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: osdmap e177: 8 total, 8 up, 8 in 2024-03-30T14:01:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]: dispatch 2024-03-30T14:01:15.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: pgmap v221: 300 pgs: 64 unknown, 236 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]': finished 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]: dispatch 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: osdmap e177: 8 total, 8 up, 8 in 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]: dispatch 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: pgmap v221: 300 pgs: 64 unknown, 236 active+clean; 607 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:15.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-15"}]': finished 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi102-74805-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: osdmap e177: 8 total, 8 up, 8 in 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:15.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:16.817 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:01:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:16 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:16 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]': finished 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]': finished 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-15", "mode": "writeback"}]': finished 2024-03-30T14:01:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: pgmap v224: 268 pgs: 268 active+clean; 615 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-30T14:01:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: osdmap e178: 8 total, 8 up, 8 in 2024-03-30T14:01:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2793344392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi102-74825-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2793344392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi102-74825-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: osdmap e179: 8 total, 8 up, 8 in 2024-03-30T14:01:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: pgmap v224: 268 pgs: 268 active+clean; 615 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: osdmap e178: 8 total, 8 up, 8 in 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2793344392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi102-74825-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2793344392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi102-74825-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: osdmap e179: 8 total, 8 up, 8 in 2024-03-30T14:01:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: pgmap v224: 268 pgs: 268 active+clean; 615 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: osdmap e178: 8 total, 8 up, 8 in 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2793344392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi102-74825-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi102-74805-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2793344392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi102-74825-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: osdmap e179: 8 total, 8 up, 8 in 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:18.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:18.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:18.567 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-30T14:01:18.567 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-30T14:01:18.567 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (11508 ms) 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (23776 ms) 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8091 ms) 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3201 ms) 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2724 ms) 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3103 ms) 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-30T14:01:18.568 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3030 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2862 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3003 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3019 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3004 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2981 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3040 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3010 ms) 2024-03-30T14:01:18.569 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3039 ms) 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3021 ms) 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3257 ms) 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2634 ms) 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3009 ms) 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3020 ms) 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3002 ms) 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-30T14:01:18.570 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3265 ms) 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2758 ms) 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3261 ms) 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2697 ms) 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3003 ms) 2024-03-30T14:01:18.571 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-30T14:01:19.586 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAishotsECPP.SnapRemovePP (1998 ms) 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2021 ms) 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1995 ms) 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8892 ms total) 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4028 ms) 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3932 ms) 2024-03-30T14:01:19.587 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4249 ms) 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12209 ms total) 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (116245 ms total) 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-30T14:01:19.588 INFO:tasks.workunit.client.0.smithi102.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-30T14:01:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: pgmap v227: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:01:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:19.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: osdmap e180: 8 total, 8 up, 8 in 2024-03-30T14:01:19.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:19.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:19.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:19.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: pgmap v227: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: osdmap e180: 8 total, 8 up, 8 in 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: pgmap v227: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:01:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:19.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:19.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: osdmap e180: 8 total, 8 up, 8 in 2024-03-30T14:01:19.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:19.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3444560391' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:19.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]: dispatch 2024-03-30T14:01:19.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]: dispatch 2024-03-30T14:01:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:20 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]': finished 2024-03-30T14:01:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:20 smithi138 ceph-mon[33397]: osdmap e181: 8 total, 8 up, 8 in 2024-03-30T14:01:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4127606278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi102-74825-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:20.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:20.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:20.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:20.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]': finished 2024-03-30T14:01:20.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:20.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[29392]: osdmap e181: 8 total, 8 up, 8 in 2024-03-30T14:01:20.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4127606278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi102-74825-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:20.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:20.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-15"}]': finished 2024-03-30T14:01:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi102-75095-21"}]': finished 2024-03-30T14:01:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[37717]: osdmap e181: 8 total, 8 up, 8 in 2024-03-30T14:01:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4127606278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi102-74825-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:21.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: pgmap v230: 292 pgs: 32 creating+peering, 260 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:21.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4127606278' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi102-74825-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:21.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:21.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: osdmap e182: 8 total, 8 up, 8 in 2024-03-30T14:01:21.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:21.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:21.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:21.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:21.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: osdmap e183: 8 total, 8 up, 8 in 2024-03-30T14:01:21.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:21.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: pgmap v230: 292 pgs: 32 creating+peering, 260 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4127606278' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi102-74825-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: osdmap e182: 8 total, 8 up, 8 in 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: osdmap e183: 8 total, 8 up, 8 in 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:21.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: pgmap v230: 292 pgs: 32 creating+peering, 260 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4127606278' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi102-74825-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: osdmap e182: 8 total, 8 up, 8 in 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1433363160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]: dispatch 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi102-74805-29"}]': finished 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: osdmap e183: 8 total, 8 up, 8 in 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:21.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:22.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: osdmap e184: 8 total, 8 up, 8 in 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1165883778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi102-74825-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:22.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: osdmap e184: 8 total, 8 up, 8 in 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1165883778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi102-74825-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi102-74805-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: osdmap e184: 8 total, 8 up, 8 in 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1165883778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi102-74825-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:22.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:23 smithi138 ceph-mon[33397]: pgmap v233: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1165883778' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi102-74825-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:24.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:23 smithi102 ceph-mon[29392]: pgmap v233: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:24.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1165883778' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi102-74825-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:24.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:23 smithi102 ceph-mon[37717]: pgmap v233: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:24.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1165883778' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi102-74825-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: osdmap e185: 8 total, 8 up, 8 in 2024-03-30T14:01:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]': finished 2024-03-30T14:01:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]: dispatch 2024-03-30T14:01:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: osdmap e186: 8 total, 8 up, 8 in 2024-03-30T14:01:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]: dispatch 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: osdmap e185: 8 total, 8 up, 8 in 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]': finished 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]: dispatch 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: osdmap e186: 8 total, 8 up, 8 in 2024-03-30T14:01:25.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]: dispatch 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: osdmap e185: 8 total, 8 up, 8 in 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi102-74805-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-17"}]': finished 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]: dispatch 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: osdmap e186: 8 total, 8 up, 8 in 2024-03-30T14:01:25.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]: dispatch 2024-03-30T14:01:25.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:01:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-mon[33397]: pgmap v236: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]': finished 2024-03-30T14:01:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-mon[33397]: osdmap e187: 8 total, 8 up, 8 in 2024-03-30T14:01:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/684126694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi102-74825-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:26.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[29392]: pgmap v236: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]': finished 2024-03-30T14:01:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[29392]: osdmap e187: 8 total, 8 up, 8 in 2024-03-30T14:01:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/684126694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi102-74825-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[37717]: pgmap v236: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-17", "mode": "writeback"}]': finished 2024-03-30T14:01:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[37717]: osdmap e187: 8 total, 8 up, 8 in 2024-03-30T14:01:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/684126694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi102-74825-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:26.852 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:01:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:27.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:27.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:27.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:27.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:27.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/684126694' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi102-74825-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:27.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:27.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: osdmap e188: 8 total, 8 up, 8 in 2024-03-30T14:01:27.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:27.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:27.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:27.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/684126694' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi102-74825-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: osdmap e188: 8 total, 8 up, 8 in 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:27.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/684126694' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi102-74825-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: osdmap e188: 8 total, 8 up, 8 in 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:27.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]: dispatch 2024-03-30T14:01:27.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:27.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:28.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:27 smithi138 ceph-mon[33397]: pgmap v239: 300 pgs: 32 unknown, 8 creating+peering, 260 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:28.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:27 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:28.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]': finished 2024-03-30T14:01:28.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:28.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:27 smithi138 ceph-mon[33397]: osdmap e189: 8 total, 8 up, 8 in 2024-03-30T14:01:28.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:28.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:28.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[29392]: pgmap v239: 300 pgs: 32 unknown, 8 creating+peering, 260 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:28.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:28.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]': finished 2024-03-30T14:01:28.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:28.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[29392]: osdmap e189: 8 total, 8 up, 8 in 2024-03-30T14:01:28.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:28.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[37717]: pgmap v239: 300 pgs: 32 unknown, 8 creating+peering, 260 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-17"}]': finished 2024-03-30T14:01:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[37717]: osdmap e189: 8 total, 8 up, 8 in 2024-03-30T14:01:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2892957868' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:28.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]: dispatch 2024-03-30T14:01:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: pgmap v242: 260 pgs: 260 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: osdmap e190: 8 total, 8 up, 8 in 2024-03-30T14:01:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi102-74825-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: pgmap v242: 260 pgs: 260 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: osdmap e190: 8 total, 8 up, 8 in 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi102-74825-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: pgmap v242: 260 pgs: 260 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi102-74805-30"}]': finished 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: osdmap e190: 8 total, 8 up, 8 in 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi102-74825-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:29.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:30.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi102-74825-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:30.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:30.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: osdmap e191: 8 total, 8 up, 8 in 2024-03-30T14:01:30.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:30.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd=[{ 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: "prefix": "osd pool set", 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: "pool": "PoolEIOFlag_smithi102-74825-33", 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: "var": "eio", 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: "val": "true" 2024-03-30T14:01:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:30 smithi138 ceph-mon[33397]: }]: dispatch 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi102-74825-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: osdmap e191: 8 total, 8 up, 8 in 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:30.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd=[{ 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: "prefix": "osd pool set", 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: "pool": "PoolEIOFlag_smithi102-74825-33", 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: "var": "eio", 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: "val": "true" 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[29392]: }]: dispatch 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi102-74825-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi102-74805-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: osdmap e191: 8 total, 8 up, 8 in 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd=[{ 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: "prefix": "osd pool set", 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: "pool": "PoolEIOFlag_smithi102-74825-33", 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: "var": "eio", 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: "val": "true" 2024-03-30T14:01:30.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:30 smithi102 ceph-mon[37717]: }]: dispatch 2024-03-30T14:01:31.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: pgmap v245: 292 pgs: 32 creating+peering, 260 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-03-30T14:01:31.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:31.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd='[{ 2024-03-30T14:01:31.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: "prefix": "osd pool set", 2024-03-30T14:01:31.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: "pool": "PoolEIOFlag_smithi102-74825-33", 2024-03-30T14:01:31.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: "var": "eio", 2024-03-30T14:01:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: "val": "true" 2024-03-30T14:01:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: }]': finished 2024-03-30T14:01:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: osdmap e192: 8 total, 8 up, 8 in 2024-03-30T14:01:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:31.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:31 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: pgmap v245: 292 pgs: 32 creating+peering, 260 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd='[{ 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: "prefix": "osd pool set", 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: "pool": "PoolEIOFlag_smithi102-74825-33", 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: "var": "eio", 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: "val": "true" 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: }]': finished 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: osdmap e192: 8 total, 8 up, 8 in 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:31.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: pgmap v245: 292 pgs: 32 creating+peering, 260 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2039704305' entity='client.admin' cmd='[{ 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: "prefix": "osd pool set", 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: "pool": "PoolEIOFlag_smithi102-74825-33", 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: "var": "eio", 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: "val": "true" 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: }]': finished 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: osdmap e192: 8 total, 8 up, 8 in 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:31.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:31 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:32.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:32.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:32.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:32.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:32 smithi138 ceph-mon[33397]: osdmap e193: 8 total, 8 up, 8 in 2024-03-30T14:01:32.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[37717]: osdmap e193: 8 total, 8 up, 8 in 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi102-74805-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[29392]: osdmap e193: 8 total, 8 up, 8 in 2024-03-30T14:01:32.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:33 smithi138 ceph-mon[33397]: pgmap v248: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]': finished 2024-03-30T14:01:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:33 smithi138 ceph-mon[33397]: osdmap e194: 8 total, 8 up, 8 in 2024-03-30T14:01:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]: dispatch 2024-03-30T14:01:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]: dispatch 2024-03-30T14:01:33.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1014304061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:33.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[29392]: pgmap v248: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]': finished 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[29392]: osdmap e194: 8 total, 8 up, 8 in 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]: dispatch 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]: dispatch 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1014304061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[37717]: pgmap v248: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-19"}]': finished 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[37717]: osdmap e194: 8 total, 8 up, 8 in 2024-03-30T14:01:33.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]: dispatch 2024-03-30T14:01:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]: dispatch 2024-03-30T14:01:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1014304061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:33.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:34.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:34.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]': finished 2024-03-30T14:01:34.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:34.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: osdmap e195: 8 total, 8 up, 8 in 2024-03-30T14:01:34.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]': finished 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: osdmap e195: 8 total, 8 up, 8 in 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-19", "mode": "writeback"}]': finished 2024-03-30T14:01:34.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi102-74825-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: osdmap e195: 8 total, 8 up, 8 in 2024-03-30T14:01:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:35.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:01:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: pgmap v251: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:35.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: osdmap e196: 8 total, 8 up, 8 in 2024-03-30T14:01:35.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:35.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:35.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:35.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: pgmap v251: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: osdmap e196: 8 total, 8 up, 8 in 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: pgmap v251: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:35.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: osdmap e196: 8 total, 8 up, 8 in 2024-03-30T14:01:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]: dispatch 2024-03-30T14:01:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1703595331' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:35.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]: dispatch 2024-03-30T14:01:36.891 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:01:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]': finished 2024-03-30T14:01:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: osdmap e197: 8 total, 8 up, 8 in 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1228538037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi102-74825-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1228538037' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi102-74825-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: osdmap e198: 8 total, 8 up, 8 in 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]': finished 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: osdmap e197: 8 total, 8 up, 8 in 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1228538037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi102-74825-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:37.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1228538037' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi102-74825-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: osdmap e198: 8 total, 8 up, 8 in 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-19"}]': finished 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi102-74805-31"}]': finished 2024-03-30T14:01:37.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: osdmap e197: 8 total, 8 up, 8 in 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1228538037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi102-74825-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1228538037' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi102-74825-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi102-74805-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: osdmap e198: 8 total, 8 up, 8 in 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:37.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:37 smithi138 ceph-mon[33397]: pgmap v254: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:01:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:37 smithi138 ceph-mon[33397]: osdmap e199: 8 total, 8 up, 8 in 2024-03-30T14:01:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[29392]: pgmap v254: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[29392]: osdmap e199: 8 total, 8 up, 8 in 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[37717]: pgmap v254: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[37717]: osdmap e199: 8 total, 8 up, 8 in 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: pgmap v257: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: osdmap e200: 8 total, 8 up, 8 in 2024-03-30T14:01:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/906055673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:39.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: pgmap v257: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: osdmap e200: 8 total, 8 up, 8 in 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/906055673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: pgmap v257: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi102-74805-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: osdmap e200: 8 total, 8 up, 8 in 2024-03-30T14:01:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/906055673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: osdmap e201: 8 total, 8 up, 8 in 2024-03-30T14:01:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]': finished 2024-03-30T14:01:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]: dispatch 2024-03-30T14:01:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: osdmap e202: 8 total, 8 up, 8 in 2024-03-30T14:01:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3869433208' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: osdmap e201: 8 total, 8 up, 8 in 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]': finished 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]: dispatch 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: osdmap e202: 8 total, 8 up, 8 in 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3869433208' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:40.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: osdmap e201: 8 total, 8 up, 8 in 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-21"}]': finished 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]: dispatch 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: osdmap e202: 8 total, 8 up, 8 in 2024-03-30T14:01:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3869433208' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: pgmap v260: 300 pgs: 300 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]: dispatch 2024-03-30T14:01:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]': finished 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: osdmap e203: 8 total, 8 up, 8 in 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: pgmap v260: 300 pgs: 300 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]: dispatch 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]': finished 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: osdmap e203: 8 total, 8 up, 8 in 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: pgmap v260: 300 pgs: 300 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]: dispatch 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-21", "mode": "writeback"}]': finished 2024-03-30T14:01:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: osdmap e203: 8 total, 8 up, 8 in 2024-03-30T14:01:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1459807157' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]: dispatch 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: osdmap e204: 8 total, 8 up, 8 in 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/786024831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:42.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: osdmap e204: 8 total, 8 up, 8 in 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/786024831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi102-74805-32"}]': finished 2024-03-30T14:01:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: osdmap e204: 8 total, 8 up, 8 in 2024-03-30T14:01:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/786024831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]: dispatch 2024-03-30T14:01:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: pgmap v263: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:01:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/786024831' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]': finished 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: osdmap e205: 8 total, 8 up, 8 in 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: pgmap v263: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:01:43.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:43.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/786024831' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]': finished 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: osdmap e205: 8 total, 8 up, 8 in 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: pgmap v263: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/786024831' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi102-74825-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-21"}]': finished 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi102-74805-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: osdmap e205: 8 total, 8 up, 8 in 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:43.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:45 smithi138 ceph-mon[33397]: pgmap v266: 356 pgs: 64 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:45 smithi138 ceph-mon[33397]: osdmap e206: 8 total, 8 up, 8 in 2024-03-30T14:01:45.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:01:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:01:45.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:45 smithi102 ceph-mon[29392]: pgmap v266: 356 pgs: 64 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:45.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:45 smithi102 ceph-mon[29392]: osdmap e206: 8 total, 8 up, 8 in 2024-03-30T14:01:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:45 smithi102 ceph-mon[37717]: pgmap v266: 356 pgs: 64 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:45 smithi102 ceph-mon[37717]: osdmap e206: 8 total, 8 up, 8 in 2024-03-30T14:01:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:46 smithi138 ceph-mon[33397]: osdmap e207: 8 total, 8 up, 8 in 2024-03-30T14:01:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:46 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:46.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:01:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[29392]: osdmap e207: 8 total, 8 up, 8 in 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi102-74805-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[37717]: osdmap e207: 8 total, 8 up, 8 in 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:46.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:46.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:46 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:47.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:47 smithi138 ceph-mon[33397]: pgmap v269: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:01:47.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:47.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:47 smithi138 ceph-mon[33397]: osdmap e208: 8 total, 8 up, 8 in 2024-03-30T14:01:47.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:47 smithi102 ceph-mon[29392]: pgmap v269: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:01:47.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:47.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:47 smithi102 ceph-mon[29392]: osdmap e208: 8 total, 8 up, 8 in 2024-03-30T14:01:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:47 smithi102 ceph-mon[37717]: pgmap v269: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:01:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:47.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:47 smithi102 ceph-mon[37717]: osdmap e208: 8 total, 8 up, 8 in 2024-03-30T14:01:48.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:48 smithi138 ceph-mon[33397]: osdmap e209: 8 total, 8 up, 8 in 2024-03-30T14:01:48.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458969564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:48.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:48.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:48.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:48.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:48.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[29392]: osdmap e209: 8 total, 8 up, 8 in 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458969564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[37717]: osdmap e209: 8 total, 8 up, 8 in 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458969564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:48.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: pgmap v272: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458969564' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: osdmap e210: 8 total, 8 up, 8 in 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:49.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: pgmap v272: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458969564' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: osdmap e210: 8 total, 8 up, 8 in 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:49.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: pgmap v272: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:01:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458969564' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: osdmap e210: 8 total, 8 up, 8 in 2024-03-30T14:01:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/152649525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]: dispatch 2024-03-30T14:01:50.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:50.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:51.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:51.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]': finished 2024-03-30T14:01:51.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]: dispatch 2024-03-30T14:01:51.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: osdmap e211: 8 total, 8 up, 8 in 2024-03-30T14:01:51.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/605808420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:51.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]': finished 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: osdmap e212: 8 total, 8 up, 8 in 2024-03-30T14:01:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:51.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]': finished 2024-03-30T14:01:51.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]: dispatch 2024-03-30T14:01:51.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: osdmap e211: 8 total, 8 up, 8 in 2024-03-30T14:01:51.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/605808420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:51.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]': finished 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: osdmap e212: 8 total, 8 up, 8 in 2024-03-30T14:01:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi102-74805-33"}]': finished 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-23"}]': finished 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: osdmap e211: 8 total, 8 up, 8 in 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/605808420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:51.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:01:51.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:51.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-23", "mode": "writeback"}]': finished 2024-03-30T14:01:51.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi102-74805-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:51.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: osdmap e212: 8 total, 8 up, 8 in 2024-03-30T14:01:51.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:51 smithi102 ceph-mon[29392]: pgmap v275: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:01:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:51 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:52.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:51 smithi102 ceph-mon[37717]: pgmap v275: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:01:52.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:52.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:51 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:52.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:51 smithi138 ceph-mon[33397]: pgmap v275: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:01:52.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:52.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:51 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:53.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: osdmap e213: 8 total, 8 up, 8 in 2024-03-30T14:01:53.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3006801166' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:53.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:53.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:53.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:53.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3006801166' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:53.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:53.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:53.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: osdmap e214: 8 total, 8 up, 8 in 2024-03-30T14:01:53.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: osdmap e213: 8 total, 8 up, 8 in 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3006801166' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3006801166' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: osdmap e214: 8 total, 8 up, 8 in 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:53.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: osdmap e213: 8 total, 8 up, 8 in 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3006801166' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi102-74805-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3006801166' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: osdmap e214: 8 total, 8 up, 8 in 2024-03-30T14:01:53.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]: dispatch 2024-03-30T14:01:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:53 smithi138 ceph-mon[33397]: pgmap v278: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T14:01:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:53 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]': finished 2024-03-30T14:01:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:53 smithi138 ceph-mon[33397]: osdmap e215: 8 total, 8 up, 8 in 2024-03-30T14:01:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1220031541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[29392]: pgmap v278: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]': finished 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[29392]: osdmap e215: 8 total, 8 up, 8 in 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1220031541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[37717]: pgmap v278: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-23"}]': finished 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[37717]: osdmap e215: 8 total, 8 up, 8 in 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1220031541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:55.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:01:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:01:55.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:55 smithi138 ceph-mon[33397]: pgmap v281: 396 pgs: 72 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:55 smithi138 ceph-mon[33397]: osdmap e216: 8 total, 8 up, 8 in 2024-03-30T14:01:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[29392]: pgmap v281: 396 pgs: 72 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[29392]: osdmap e216: 8 total, 8 up, 8 in 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[37717]: pgmap v281: 396 pgs: 72 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[37717]: osdmap e216: 8 total, 8 up, 8 in 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:55.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:56.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:01:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:01:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: osdmap e217: 8 total, 8 up, 8 in 2024-03-30T14:01:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4250411090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: osdmap e217: 8 total, 8 up, 8 in 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:56.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4250411090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:56.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:56.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:56.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:56 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: osdmap e217: 8 total, 8 up, 8 in 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2601631218' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4250411090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]: dispatch 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:01:57.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:01:57.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:56 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: pgmap v284: 420 pgs: 64 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: osdmap e218: 8 total, 8 up, 8 in 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:58.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi102-74805-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: pgmap v284: 420 pgs: 64 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: osdmap e218: 8 total, 8 up, 8 in 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi102-74805-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: pgmap v284: 420 pgs: 64 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi102-74805-34"}]': finished 2024-03-30T14:01:58.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi102-74825-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:01:58.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: osdmap e218: 8 total, 8 up, 8 in 2024-03-30T14:01:58.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:58.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:58.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi102-74805-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:01:59.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi102-74805-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:59.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:58 smithi138 ceph-mon[33397]: osdmap e219: 8 total, 8 up, 8 in 2024-03-30T14:01:59.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi102-74805-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:59.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:59.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:59.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:59.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi102-74805-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:59.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[29392]: osdmap e219: 8 total, 8 up, 8 in 2024-03-30T14:01:59.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi102-74805-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:59.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:59.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:59.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:01:59.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi102-74805-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:01:59.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[37717]: osdmap e219: 8 total, 8 up, 8 in 2024-03-30T14:01:59.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi102-74805-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:01:59.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:59.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:01:59.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: pgmap v287: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:02:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: osdmap e220: 8 total, 8 up, 8 in 2024-03-30T14:02:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi102-74805-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]': finished 2024-03-30T14:02:00.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]: dispatch 2024-03-30T14:02:00.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: osdmap e221: 8 total, 8 up, 8 in 2024-03-30T14:02:00.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:01:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]: dispatch 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: pgmap v287: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: osdmap e220: 8 total, 8 up, 8 in 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi102-74805-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]': finished 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]: dispatch 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: osdmap e221: 8 total, 8 up, 8 in 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]: dispatch 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: pgmap v287: 388 pgs: 32 unknown, 356 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:02:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:00.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: osdmap e220: 8 total, 8 up, 8 in 2024-03-30T14:02:00.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:00.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi102-74805-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:00.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-25"}]': finished 2024-03-30T14:02:00.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]: dispatch 2024-03-30T14:02:00.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: osdmap e221: 8 total, 8 up, 8 in 2024-03-30T14:02:00.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:01:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]: dispatch 2024-03-30T14:02:01.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:00 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:01.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]': finished 2024-03-30T14:02:01.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:00 smithi138 ceph-mon[33397]: osdmap e222: 8 total, 8 up, 8 in 2024-03-30T14:02:01.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:01.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:01.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:01.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]': finished 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[29392]: osdmap e222: 8 total, 8 up, 8 in 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-25", "mode": "writeback"}]': finished 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[37717]: osdmap e222: 8 total, 8 up, 8 in 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:01.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:01 smithi138 ceph-mon[33397]: pgmap v290: 332 pgs: 8 creating+peering, 324 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:02:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:01 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:01 smithi138 ceph-mon[33397]: osdmap e223: 8 total, 8 up, 8 in 2024-03-30T14:02:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[29392]: pgmap v290: 332 pgs: 8 creating+peering, 324 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[29392]: osdmap e223: 8 total, 8 up, 8 in 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:02:02.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[37717]: pgmap v290: 332 pgs: 8 creating+peering, 324 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:02:02.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:02.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:02.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[37717]: osdmap e223: 8 total, 8 up, 8 in 2024-03-30T14:02:02.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:02.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]: dispatch 2024-03-30T14:02:02.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:02:03.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:02 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:03.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]': finished 2024-03-30T14:02:03.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:03.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:02 smithi138 ceph-mon[33397]: osdmap e224: 8 total, 8 up, 8 in 2024-03-30T14:02:03.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:02:03.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/86836382' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:03.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:03.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]': finished 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[29392]: osdmap e224: 8 total, 8 up, 8 in 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/86836382' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-25"}]': finished 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[37717]: osdmap e224: 8 total, 8 up, 8 in 2024-03-30T14:02:03.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]: dispatch 2024-03-30T14:02:03.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/86836382' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:03.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:04.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: pgmap v293: 260 pgs: 260 active+clean; 583 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:04.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:04.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:04.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: osdmap e225: 8 total, 8 up, 8 in 2024-03-30T14:02:04.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:04.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:04.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: pgmap v293: 260 pgs: 260 active+clean; 583 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: osdmap e225: 8 total, 8 up, 8 in 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:04.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: pgmap v293: 260 pgs: 260 active+clean; 583 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2536436464' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi102-74805-35"}]': finished 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi102-74825-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: osdmap e225: 8 total, 8 up, 8 in 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:04.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:05.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: pgmap v296: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: osdmap e226: 8 total, 8 up, 8 in 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:05 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:02:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: pgmap v296: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: osdmap e226: 8 total, 8 up, 8 in 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: pgmap v296: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi102-74805-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: osdmap e226: 8 total, 8 up, 8 in 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:06.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:05 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:02:06.853 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:02:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: osdmap e227: 8 total, 8 up, 8 in 2024-03-30T14:02:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1216816408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi102-74825-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1216816408' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi102-74825-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: osdmap e228: 8 total, 8 up, 8 in 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:07.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: osdmap e227: 8 total, 8 up, 8 in 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1216816408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi102-74825-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1216816408' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi102-74825-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: osdmap e228: 8 total, 8 up, 8 in 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: osdmap e227: 8 total, 8 up, 8 in 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1216816408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi102-74825-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi102-74805-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1216816408' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi102-74825-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:07.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: osdmap e228: 8 total, 8 up, 8 in 2024-03-30T14:02:07.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:07.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: pgmap v299: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: osdmap e229: 8 total, 8 up, 8 in 2024-03-30T14:02:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:08.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: pgmap v299: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: osdmap e229: 8 total, 8 up, 8 in 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:08.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: pgmap v299: 292 pgs: 32 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: osdmap e229: 8 total, 8 up, 8 in 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: pgmap v302: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]': finished 2024-03-30T14:02:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: osdmap e230: 8 total, 8 up, 8 in 2024-03-30T14:02:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]: dispatch 2024-03-30T14:02:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]: dispatch 2024-03-30T14:02:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:10.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:10.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: pgmap v302: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:10.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]': finished 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: osdmap e230: 8 total, 8 up, 8 in 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]: dispatch 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]: dispatch 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: pgmap v302: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 583 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-27"}]': finished 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: osdmap e230: 8 total, 8 up, 8 in 2024-03-30T14:02:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]: dispatch 2024-03-30T14:02:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]: dispatch 2024-03-30T14:02:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]': finished 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: osdmap e231: 8 total, 8 up, 8 in 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: osdmap e232: 8 total, 8 up, 8 in 2024-03-30T14:02:11.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:11.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:11.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:11.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:11.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:11.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:11.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]': finished 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: osdmap e231: 8 total, 8 up, 8 in 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: osdmap e232: 8 total, 8 up, 8 in 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-27", "mode": "writeback"}]': finished 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3593730289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: osdmap e231: 8 total, 8 up, 8 in 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]: dispatch 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi102-74805-36"}]': finished 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: osdmap e232: 8 total, 8 up, 8 in 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:11.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:11.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: pgmap v305: 260 pgs: 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: osdmap e233: 8 total, 8 up, 8 in 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi102-74805-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: pgmap v305: 260 pgs: 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: osdmap e233: 8 total, 8 up, 8 in 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi102-74805-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: pgmap v305: 260 pgs: 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:12.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:12.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:12.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:12.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: osdmap e233: 8 total, 8 up, 8 in 2024-03-30T14:02:12.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi102-74805-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:12.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]: dispatch 2024-03-30T14:02:13.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:12 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:13.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]': finished 2024-03-30T14:02:13.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:12 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:13.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]': finished 2024-03-30T14:02:13.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:12 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:13.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-27"}]': finished 2024-03-30T14:02:14.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: pgmap v308: 268 pgs: 8 unknown, 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:02:14.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: osdmap e234: 8 total, 8 up, 8 in 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi102-74805-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: osdmap e235: 8 total, 8 up, 8 in 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.295 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: pgmap v308: 268 pgs: 8 unknown, 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: osdmap e234: 8 total, 8 up, 8 in 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi102-74805-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:14.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: osdmap e235: 8 total, 8 up, 8 in 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: pgmap v308: 268 pgs: 8 unknown, 260 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: osdmap e234: 8 total, 8 up, 8 in 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi102-74805-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: osdmap e235: 8 total, 8 up, 8 in 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1093064256' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:14.297 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]: dispatch 2024-03-30T14:02:15.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:02:15.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: pgmap v311: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:15.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:15.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: osdmap e236: 8 total, 8 up, 8 in 2024-03-30T14:02:15.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:15.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:15.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: pgmap v311: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:15.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: osdmap e236: 8 total, 8 up, 8 in 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: pgmap v311: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-46"}]': finished 2024-03-30T14:02:15.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: osdmap e236: 8 total, 8 up, 8 in 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:15.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:16.854 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:02:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: osdmap e237: 8 total, 8 up, 8 in 2024-03-30T14:02:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:17.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: osdmap e237: 8 total, 8 up, 8 in 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi102-74825-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: osdmap e237: 8 total, 8 up, 8 in 2024-03-30T14:02:17.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:17.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:17.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:17.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: pgmap v314: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: osdmap e238: 8 total, 8 up, 8 in 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: osdmap e239: 8 total, 8 up, 8 in 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:18.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: pgmap v314: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: osdmap e238: 8 total, 8 up, 8 in 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: osdmap e239: 8 total, 8 up, 8 in 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: pgmap v314: 260 pgs: 32 creating+peering, 228 active+clean; 584 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: osdmap e238: 8 total, 8 up, 8 in 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]: dispatch 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi102-74825-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3336119992' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi102-74805-37"}]': finished 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: osdmap e239: 8 total, 8 up, 8 in 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:18.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:18.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: pgmap v317: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: osdmap e240: 8 total, 8 up, 8 in 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:19 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:20.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: pgmap v317: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:20.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: osdmap e240: 8 total, 8 up, 8 in 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: pgmap v317: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 584 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi102-74805-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:20.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: osdmap e240: 8 total, 8 up, 8 in 2024-03-30T14:02:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:20.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:19 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]': finished 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]: dispatch 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: osdmap e241: 8 total, 8 up, 8 in 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]: dispatch 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]': finished 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:21.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: osdmap e242: 8 total, 8 up, 8 in 2024-03-30T14:02:21.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]': finished 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]: dispatch 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: osdmap e241: 8 total, 8 up, 8 in 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]: dispatch 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]': finished 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: osdmap e242: 8 total, 8 up, 8 in 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-29"}]': finished 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]: dispatch 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: osdmap e241: 8 total, 8 up, 8 in 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]: dispatch 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi102-74805-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-29", "mode": "writeback"}]': finished 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: osdmap e242: 8 total, 8 up, 8 in 2024-03-30T14:02:21.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/7051313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:21.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]: dispatch 2024-03-30T14:02:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:21 smithi138 ceph-mon[33397]: pgmap v320: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:21 smithi138 ceph-mon[33397]: osdmap e243: 8 total, 8 up, 8 in 2024-03-30T14:02:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi102-74825-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[29392]: pgmap v320: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[29392]: osdmap e243: 8 total, 8 up, 8 in 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi102-74825-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[37717]: pgmap v320: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi102-74825-47"}]': finished 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[37717]: osdmap e243: 8 total, 8 up, 8 in 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi102-74825-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:23 smithi138 ceph-mon[33397]: pgmap v323: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi102-74825-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:23 smithi138 ceph-mon[33397]: osdmap e244: 8 total, 8 up, 8 in 2024-03-30T14:02:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi102-74825-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:24.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:24.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[29392]: pgmap v323: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi102-74825-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[29392]: osdmap e244: 8 total, 8 up, 8 in 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi102-74825-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[37717]: pgmap v323: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi102-74825-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:24.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[37717]: osdmap e244: 8 total, 8 up, 8 in 2024-03-30T14:02:24.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi102-74825-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:24.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:24.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:25.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: osdmap e245: 8 total, 8 up, 8 in 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]: dispatch 2024-03-30T14:02:25.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]: dispatch 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]: dispatch 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi102-74825-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:25.090 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: osdmap e245: 8 total, 8 up, 8 in 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]: dispatch 2024-03-30T14:02:25.091 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]: dispatch 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi102-74825-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]': finished 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: osdmap e246: 8 total, 8 up, 8 in 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:25.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]': finished 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]': finished 2024-03-30T14:02:25.093 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]': finished 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: osdmap e246: 8 total, 8 up, 8 in 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:25.094 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2554478602' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: osdmap e245: 8 total, 8 up, 8 in 2024-03-30T14:02:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]: dispatch 2024-03-30T14:02:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:25.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]: dispatch 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi102-74825-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:25.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi102-74805-38"}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.8", "id": [6, 1]}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.9", "id": [3, 1]}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.3", "id": [5, 0]}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "242.9", "id": [5, 1]}]': finished 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: osdmap e246: 8 total, 8 up, 8 in 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]: dispatch 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:25.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:25.831 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:02:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: pgmap v326: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]': finished 2024-03-30T14:02:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: osdmap e247: 8 total, 8 up, 8 in 2024-03-30T14:02:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:26.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: pgmap v326: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]': finished 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: osdmap e247: 8 total, 8 up, 8 in 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: pgmap v326: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-29"}]': finished 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi102-74805-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: osdmap e247: 8 total, 8 up, 8 in 2024-03-30T14:02:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:26.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:26.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:02:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:02:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: pgmap v329: 268 pgs: 8 creating+peering, 6 peering, 254 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: osdmap e248: 8 total, 8 up, 8 in 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: osdmap e249: 8 total, 8 up, 8 in 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: pgmap v329: 268 pgs: 8 creating+peering, 6 peering, 254 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-30T14:02:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: osdmap e248: 8 total, 8 up, 8 in 2024-03-30T14:02:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: osdmap e249: 8 total, 8 up, 8 in 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: pgmap v329: 268 pgs: 8 creating+peering, 6 peering, 254 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: osdmap e248: 8 total, 8 up, 8 in 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T14:02:28.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:28.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi102-74805-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:28.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:28.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: osdmap e249: 8 total, 8 up, 8 in 2024-03-30T14:02:28.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]: dispatch 2024-03-30T14:02:28.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:28.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: pgmap v332: 268 pgs: 40 unknown, 4 peering, 224 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: osdmap e250: 8 total, 8 up, 8 in 2024-03-30T14:02:29.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:29.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi102-74825-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:29.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: pgmap v332: 268 pgs: 40 unknown, 4 peering, 224 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:29.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: osdmap e250: 8 total, 8 up, 8 in 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi102-74825-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: pgmap v332: 268 pgs: 40 unknown, 4 peering, 224 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-30T14:02:29.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3632674920' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi102-74825-48"}]': finished 2024-03-30T14:02:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: osdmap e250: 8 total, 8 up, 8 in 2024-03-30T14:02:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:29.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi102-74825-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi102-74825-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: osdmap e251: 8 total, 8 up, 8 in 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi102-74825-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:31.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]': finished 2024-03-30T14:02:31.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi102-74825-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: osdmap e251: 8 total, 8 up, 8 in 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi102-74825-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]': finished 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi102-74825-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:31.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: osdmap e251: 8 total, 8 up, 8 in 2024-03-30T14:02:31.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:31.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi102-74825-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:31.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:31.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:31.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:31.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-31"}]': finished 2024-03-30T14:02:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: pgmap v335: 260 pgs: 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]: dispatch 2024-03-30T14:02:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: osdmap e252: 8 total, 8 up, 8 in 2024-03-30T14:02:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]: dispatch 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi102-74825-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]': finished 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: osdmap e253: 8 total, 8 up, 8 in 2024-03-30T14:02:32.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:32.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:32.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:32.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: pgmap v335: 260 pgs: 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]: dispatch 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: osdmap e252: 8 total, 8 up, 8 in 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]: dispatch 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:32.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi102-74825-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]': finished 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: osdmap e253: 8 total, 8 up, 8 in 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:32.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: pgmap v335: 260 pgs: 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]: dispatch 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1760337262' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: osdmap e252: 8 total, 8 up, 8 in 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]: dispatch 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]: dispatch 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi102-74825-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:32.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-31", "mode": "writeback"}]': finished 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi102-74805-39"}]': finished 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: osdmap e253: 8 total, 8 up, 8 in 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:32.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: pgmap v338: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: osdmap e254: 8 total, 8 up, 8 in 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:34.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: pgmap v338: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:34.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:34.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:34.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: osdmap e254: 8 total, 8 up, 8 in 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: pgmap v338: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi102-74805-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: osdmap e254: 8 total, 8 up, 8 in 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]: dispatch 2024-03-30T14:02:35.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]': finished 2024-03-30T14:02:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: osdmap e255: 8 total, 8 up, 8 in 2024-03-30T14:02:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: osdmap e256: 8 total, 8 up, 8 in 2024-03-30T14:02:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:35.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:35.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]': finished 2024-03-30T14:02:35.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: osdmap e255: 8 total, 8 up, 8 in 2024-03-30T14:02:35.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: osdmap e256: 8 total, 8 up, 8 in 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-31"}]': finished 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: osdmap e255: 8 total, 8 up, 8 in 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi102-74805-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: osdmap e256: 8 total, 8 up, 8 in 2024-03-30T14:02:35.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]: dispatch 2024-03-30T14:02:35.844 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:02:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: pgmap v341: 260 pgs: 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: osdmap e257: 8 total, 8 up, 8 in 2024-03-30T14:02:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:36.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:36.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: pgmap v341: 260 pgs: 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:36.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:36.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: osdmap e257: 8 total, 8 up, 8 in 2024-03-30T14:02:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: pgmap v341: 260 pgs: 260 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2458753668' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi102-74825-49"}]': finished 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: osdmap e257: 8 total, 8 up, 8 in 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:36.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:36.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:36.852 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:02:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: osdmap e258: 8 total, 8 up, 8 in 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:37.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: osdmap e258: 8 total, 8 up, 8 in 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:37.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi102-74825-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: osdmap e258: 8 total, 8 up, 8 in 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:37.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:38.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: pgmap v344: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: osdmap e259: 8 total, 8 up, 8 in 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:38.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: pgmap v344: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:02:38.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:38.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:38.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: osdmap e259: 8 total, 8 up, 8 in 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: pgmap v344: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2922610441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: osdmap e259: 8 total, 8 up, 8 in 2024-03-30T14:02:38.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:38.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]: dispatch 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: pgmap v347: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]': finished 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: osdmap e260: 8 total, 8 up, 8 in 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]: dispatch 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]: dispatch 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:39.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi102-74805-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:39.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: pgmap v347: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:02:39.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:39.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]': finished 2024-03-30T14:02:39.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: osdmap e260: 8 total, 8 up, 8 in 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]: dispatch 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]: dispatch 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi102-74805-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: pgmap v347: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi102-74825-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-33"}]': finished 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi102-74805-40"}]': finished 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: osdmap e260: 8 total, 8 up, 8 in 2024-03-30T14:02:39.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]: dispatch 2024-03-30T14:02:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]: dispatch 2024-03-30T14:02:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:39.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi102-74805-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:40 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]': finished 2024-03-30T14:02:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi102-74805-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:40 smithi138 ceph-mon[33397]: osdmap e261: 8 total, 8 up, 8 in 2024-03-30T14:02:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi102-74805-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:40.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:40.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:40.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]': finished 2024-03-30T14:02:40.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi102-74805-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:40.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[29392]: osdmap e261: 8 total, 8 up, 8 in 2024-03-30T14:02:40.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi102-74805-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-33", "mode": "writeback"}]': finished 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi102-74805-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[37717]: osdmap e261: 8 total, 8 up, 8 in 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi102-74805-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:40.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: pgmap v350: 268 pgs: 268 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-30T14:02:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: osdmap e262: 8 total, 8 up, 8 in 2024-03-30T14:02:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:41.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:41.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:41 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: pgmap v350: 268 pgs: 268 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-30T14:02:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: osdmap e262: 8 total, 8 up, 8 in 2024-03-30T14:02:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: pgmap v350: 268 pgs: 268 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: osdmap e262: 8 total, 8 up, 8 in 2024-03-30T14:02:41.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]: dispatch 2024-03-30T14:02:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:41.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:41 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:42 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi102-74805-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]': finished 2024-03-30T14:02:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:42 smithi138 ceph-mon[33397]: osdmap e263: 8 total, 8 up, 8 in 2024-03-30T14:02:42.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:42.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:42.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi102-74805-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]': finished 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[29392]: osdmap e263: 8 total, 8 up, 8 in 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi102-74805-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-33"}]': finished 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[37717]: osdmap e263: 8 total, 8 up, 8 in 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2604323166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:42.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]: dispatch 2024-03-30T14:02:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:43 smithi138 ceph-mon[33397]: pgmap v353: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T14:02:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:43.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:43 smithi138 ceph-mon[33397]: osdmap e264: 8 total, 8 up, 8 in 2024-03-30T14:02:43.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:43 smithi102 ceph-mon[29392]: pgmap v353: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T14:02:43.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:43.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:43 smithi102 ceph-mon[29392]: osdmap e264: 8 total, 8 up, 8 in 2024-03-30T14:02:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:43 smithi102 ceph-mon[37717]: pgmap v353: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T14:02:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi102-74825-50"}]': finished 2024-03-30T14:02:43.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:43 smithi102 ceph-mon[37717]: osdmap e264: 8 total, 8 up, 8 in 2024-03-30T14:02:44.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:44 smithi138 ceph-mon[33397]: osdmap e265: 8 total, 8 up, 8 in 2024-03-30T14:02:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/778386776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[29392]: osdmap e265: 8 total, 8 up, 8 in 2024-03-30T14:02:44.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/778386776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[37717]: osdmap e265: 8 total, 8 up, 8 in 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/778386776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:44.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:45.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:02:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: pgmap v356: 292 pgs: 64 unknown, 228 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: osdmap e266: 8 total, 8 up, 8 in 2024-03-30T14:02:45.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:45.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:45.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:45.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: pgmap v356: 292 pgs: 64 unknown, 228 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:45.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:45.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:45.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:45.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: osdmap e266: 8 total, 8 up, 8 in 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: pgmap v356: 292 pgs: 64 unknown, 228 active+clean; 4.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi102-74825-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: osdmap e266: 8 total, 8 up, 8 in 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]: dispatch 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:45.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: osdmap e267: 8 total, 8 up, 8 in 2024-03-30T14:02:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:46.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi102-74805-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi102-74805-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:46.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: osdmap e268: 8 total, 8 up, 8 in 2024-03-30T14:02:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi102-74805-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]: dispatch 2024-03-30T14:02:46.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]: dispatch 2024-03-30T14:02:46.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:02:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: osdmap e267: 8 total, 8 up, 8 in 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi102-74805-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi102-74805-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: osdmap e268: 8 total, 8 up, 8 in 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi102-74805-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]: dispatch 2024-03-30T14:02:46.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2254670354' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi102-74805-41"}]': finished 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: osdmap e267: 8 total, 8 up, 8 in 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi102-74805-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:46.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi102-74805-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: osdmap e268: 8 total, 8 up, 8 in 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi102-74805-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]: dispatch 2024-03-30T14:02:46.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]: dispatch 2024-03-30T14:02:48.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:48 smithi138 ceph-mon[33397]: pgmap v359: 260 pgs: 260 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:02:48.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:48.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:48.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:48 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:48.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]': finished 2024-03-30T14:02:48.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[37717]: pgmap v359: 260 pgs: 260 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:02:48.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:48.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:48.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:48.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]': finished 2024-03-30T14:02:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[29392]: pgmap v359: 260 pgs: 260 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:02:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:02:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-35", "mode": "writeback"}]': finished 2024-03-30T14:02:49.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:49 smithi138 ceph-mon[33397]: osdmap e269: 8 total, 8 up, 8 in 2024-03-30T14:02:49.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi102-74805-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:49.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:49.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:49 smithi138 ceph-mon[33397]: osdmap e270: 8 total, 8 up, 8 in 2024-03-30T14:02:49.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[29392]: osdmap e269: 8 total, 8 up, 8 in 2024-03-30T14:02:49.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi102-74805-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:49.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:49.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[29392]: osdmap e270: 8 total, 8 up, 8 in 2024-03-30T14:02:49.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[37717]: osdmap e269: 8 total, 8 up, 8 in 2024-03-30T14:02:49.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi102-74805-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:49.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi102-74825-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:49.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:49 smithi102 ceph-mon[37717]: osdmap e270: 8 total, 8 up, 8 in 2024-03-30T14:02:50.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:50 smithi138 ceph-mon[33397]: pgmap v362: 260 pgs: 260 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:50.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:50 smithi138 ceph-mon[33397]: osdmap e271: 8 total, 8 up, 8 in 2024-03-30T14:02:50.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:50.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[29392]: pgmap v362: 260 pgs: 260 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[29392]: osdmap e271: 8 total, 8 up, 8 in 2024-03-30T14:02:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:50.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[37717]: pgmap v362: 260 pgs: 260 active+clean; 4.6 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:50.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[37717]: osdmap e271: 8 total, 8 up, 8 in 2024-03-30T14:02:50.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:50.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:51.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: pgmap v365: 276 pgs: 276 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-30T14:02:51.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:51.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: osdmap e272: 8 total, 8 up, 8 in 2024-03-30T14:02:51.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:51.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:51.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:51.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:51.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:51.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:51 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:51.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: pgmap v365: 276 pgs: 276 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-30T14:02:51.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:51.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: osdmap e272: 8 total, 8 up, 8 in 2024-03-30T14:02:51.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:51.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:51.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:51.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:51.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:51.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:51.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: pgmap v365: 276 pgs: 276 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-30T14:02:51.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:51.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: osdmap e272: 8 total, 8 up, 8 in 2024-03-30T14:02:51.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:51.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:51.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:51.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:51.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:51.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:51 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7023 ms) 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6955 ms) 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6997 ms) 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7019 ms) 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7047 ms) 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7018 ms) 2024-03-30T14:02:52.646 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7037 ms) 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7032 ms) 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7119 ms) 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6757 ms) 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7018 ms) 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6975 ms) 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-30T14:02:52.647 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7002 ms) 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7007 ms) 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7041 ms) 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7011 ms) 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112059 ms total) 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [----------] Global test environment tear-down 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (209405 ms total) 2024-03-30T14:02:52.648 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-30T14:02:52.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:52.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:52.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:52.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: osdmap e273: 8 total, 8 up, 8 in 2024-03-30T14:02:52.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:52.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:52.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:52.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:52.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:52.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:52 smithi138 ceph-mon[33397]: osdmap e274: 8 total, 8 up, 8 in 2024-03-30T14:02:52.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:52.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: osdmap e273: 8 total, 8 up, 8 in 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[29392]: osdmap e274: 8 total, 8 up, 8 in 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:52.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:52.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: osdmap e273: 8 total, 8 up, 8 in 2024-03-30T14:02:52.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/590635539' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:52.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]: dispatch 2024-03-30T14:02:52.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]: dispatch 2024-03-30T14:02:52.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2062261978' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi102-74805-42"}]': finished 2024-03-30T14:02:52.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi102-74825-52"}]': finished 2024-03-30T14:02:52.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:52 smithi102 ceph-mon[37717]: osdmap e274: 8 total, 8 up, 8 in 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: pgmap v368: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:53.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:53.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: pgmap v368: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:02:53.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:53.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: pgmap v368: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:53.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:02:55.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: osdmap e275: 8 total, 8 up, 8 in 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:55.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:55.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:55.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:55.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: osdmap e275: 8 total, 8 up, 8 in 2024-03-30T14:02:55.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: osdmap e275: 8 total, 8 up, 8 in 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]: dispatch 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:55.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:55.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:02:55.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-35"}]': finished 2024-03-30T14:02:55.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:02:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:02:56.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:55 smithi138 ceph-mon[33397]: pgmap v371: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:56.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:55 smithi138 ceph-mon[33397]: osdmap e276: 8 total, 8 up, 8 in 2024-03-30T14:02:56.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:02:56.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:55 smithi138 ceph-mon[33397]: osdmap e277: 8 total, 8 up, 8 in 2024-03-30T14:02:56.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[29392]: pgmap v371: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:56.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[29392]: osdmap e276: 8 total, 8 up, 8 in 2024-03-30T14:02:56.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:02:56.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[29392]: osdmap e277: 8 total, 8 up, 8 in 2024-03-30T14:02:56.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[37717]: pgmap v371: 260 pgs: 260 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:02:56.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[37717]: osdmap e276: 8 total, 8 up, 8 in 2024-03-30T14:02:56.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi102-74825-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:02:56.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:55 smithi102 ceph-mon[37717]: osdmap e277: 8 total, 8 up, 8 in 2024-03-30T14:02:56.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:02:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:02:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:56 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:56 smithi138 ceph-mon[33397]: osdmap e278: 8 total, 8 up, 8 in 2024-03-30T14:02:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:56 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:56 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[37717]: osdmap e278: 8 total, 8 up, 8 in 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[29392]: osdmap e278: 8 total, 8 up, 8 in 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:02:57.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:02:57.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:56 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:02:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:57 smithi138 ceph-mon[33397]: pgmap v374: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:57 smithi138 ceph-mon[33397]: osdmap e279: 8 total, 8 up, 8 in 2024-03-30T14:02:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[37717]: pgmap v374: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[37717]: osdmap e279: 8 total, 8 up, 8 in 2024-03-30T14:02:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[29392]: pgmap v374: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[29392]: osdmap e279: 8 total, 8 up, 8 in 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: pgmap v377: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: osdmap e280: 8 total, 8 up, 8 in 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:02:59.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:02:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:59.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: pgmap v377: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: osdmap e280: 8 total, 8 up, 8 in 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: pgmap v377: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: osdmap e280: 8 total, 8 up, 8 in 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:02:59.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1826716952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:02:59.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:02:59.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:02:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]': finished 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: osdmap e281: 8 total, 8 up, 8 in 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:00.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:00.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]': finished 2024-03-30T14:03:00.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:03:00.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]: dispatch 2024-03-30T14:03:00.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: osdmap e281: 8 total, 8 up, 8 in 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-37"}]': finished 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi102-74825-53"}]': finished 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: osdmap e281: 8 total, 8 up, 8 in 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]: dispatch 2024-03-30T14:03:00.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:00.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:00.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: pgmap v380: 260 pgs: 260 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]': finished 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: osdmap e282: 8 total, 8 up, 8 in 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:03:02.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: pgmap v380: 260 pgs: 260 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:02.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]': finished 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: osdmap e282: 8 total, 8 up, 8 in 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: pgmap v380: 260 pgs: 260 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:02.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-37", "mode": "writeback"}]': finished 2024-03-30T14:03:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi102-74825-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: osdmap e282: 8 total, 8 up, 8 in 2024-03-30T14:03:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:03:03.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:03:03.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:02 smithi138 ceph-mon[33397]: osdmap e283: 8 total, 8 up, 8 in 2024-03-30T14:03:03.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:03:03.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:02 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:03.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:03.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]': finished 2024-03-30T14:03:03.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:02 smithi138 ceph-mon[33397]: osdmap e284: 8 total, 8 up, 8 in 2024-03-30T14:03:03.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:03:03.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[29392]: osdmap e283: 8 total, 8 up, 8 in 2024-03-30T14:03:03.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:03:03.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:03.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]': finished 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[29392]: osdmap e284: 8 total, 8 up, 8 in 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[37717]: osdmap e283: 8 total, 8 up, 8 in 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]: dispatch 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi102-74825-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-37"}]': finished 2024-03-30T14:03:03.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:02 smithi102 ceph-mon[37717]: osdmap e284: 8 total, 8 up, 8 in 2024-03-30T14:03:04.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:03 smithi138 ceph-mon[33397]: pgmap v383: 260 pgs: 260 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:03 smithi138 ceph-mon[33397]: osdmap e285: 8 total, 8 up, 8 in 2024-03-30T14:03:04.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:03 smithi102 ceph-mon[37717]: pgmap v383: 260 pgs: 260 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:04.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:03 smithi102 ceph-mon[37717]: osdmap e285: 8 total, 8 up, 8 in 2024-03-30T14:03:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:03 smithi102 ceph-mon[29392]: pgmap v383: 260 pgs: 260 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:04.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:03 smithi102 ceph-mon[29392]: osdmap e285: 8 total, 8 up, 8 in 2024-03-30T14:03:05.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:03:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-mon[33397]: pgmap v386: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-mon[33397]: osdmap e286: 8 total, 8 up, 8 in 2024-03-30T14:03:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:05 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:05.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[37717]: pgmap v386: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:05.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[37717]: osdmap e286: 8 total, 8 up, 8 in 2024-03-30T14:03:05.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:05.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[29392]: pgmap v386: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[29392]: osdmap e286: 8 total, 8 up, 8 in 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:05.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:05 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:06.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: osdmap e287: 8 total, 8 up, 8 in 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:06.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:03:06.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:06.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:03:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: osdmap e287: 8 total, 8 up, 8 in 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:03:06.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: osdmap e287: 8 total, 8 up, 8 in 2024-03-30T14:03:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3201933774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]: dispatch 2024-03-30T14:03:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:06.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:06.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:03:06.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi102-74825-54"}]': finished 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: pgmap v389: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: osdmap e288: 8 total, 8 up, 8 in 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi102-74825-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:07.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]': finished 2024-03-30T14:03:07.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi102-74825-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:07.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: osdmap e289: 8 total, 8 up, 8 in 2024-03-30T14:03:07.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi102-74825-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]: dispatch 2024-03-30T14:03:07.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]: dispatch 2024-03-30T14:03:07.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: pgmap v389: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:03:07.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:07.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: osdmap e288: 8 total, 8 up, 8 in 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi102-74825-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]': finished 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi102-74825-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: osdmap e289: 8 total, 8 up, 8 in 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi102-74825-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]: dispatch 2024-03-30T14:03:07.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]: dispatch 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: pgmap v389: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: osdmap e288: 8 total, 8 up, 8 in 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi102-74825-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:07.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:07.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-39"}]': finished 2024-03-30T14:03:07.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi102-74825-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:07.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: osdmap e289: 8 total, 8 up, 8 in 2024-03-30T14:03:07.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi102-74825-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:07.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]: dispatch 2024-03-30T14:03:07.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]: dispatch 2024-03-30T14:03:09.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:08 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:03:09.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:08 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:03:09.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:08 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:09.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:08 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:09.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]': finished 2024-03-30T14:03:09.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:03:09.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:03:09.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:09.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:09.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]': finished 2024-03-30T14:03:09.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:03:09.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:03:09.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:09.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:09.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-39", "mode": "writeback"}]': finished 2024-03-30T14:03:10.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: pgmap v392: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: osdmap e290: 8 total, 8 up, 8 in 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi102-74825-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: osdmap e291: 8 total, 8 up, 8 in 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:10.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:10.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: pgmap v392: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:03:10.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:10.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: osdmap e290: 8 total, 8 up, 8 in 2024-03-30T14:03:10.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi102-74825-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: osdmap e291: 8 total, 8 up, 8 in 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: pgmap v392: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: osdmap e290: 8 total, 8 up, 8 in 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi102-74825-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:10.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:03:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: osdmap e291: 8 total, 8 up, 8 in 2024-03-30T14:03:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:10.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]: dispatch 2024-03-30T14:03:11.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:10 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]': finished 2024-03-30T14:03:11.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:10 smithi138 ceph-mon[33397]: osdmap e292: 8 total, 8 up, 8 in 2024-03-30T14:03:11.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:10 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:11.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]': finished 2024-03-30T14:03:11.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:10 smithi102 ceph-mon[29392]: osdmap e292: 8 total, 8 up, 8 in 2024-03-30T14:03:11.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:10 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:11.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-39"}]': finished 2024-03-30T14:03:11.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:10 smithi102 ceph-mon[37717]: osdmap e292: 8 total, 8 up, 8 in 2024-03-30T14:03:11.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-30T14:03:11.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-30T14:03:11.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-30T14:03:11.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: seed 75117 2024-03-30T14:03:11.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-30T14:03:11.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (345 ms) 2024-03-30T14:03:11.596 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (18864 ms) 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (162 ms) 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (19371 ms total) 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (9424 ms) 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8027 ms) 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9912 ms) 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-30T14:03:11.597 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: my_snaps [3] 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: my_snaps [4,3] 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: promoting some heads 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: waiting for scrubs... 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: done waiting 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46076 ms) 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-30T14:03:11.598 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9963 ms) 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7987 ms) 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8054 ms) 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8269 ms) 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9792 ms) 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8885 ms) 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10022 ms) 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-30T14:03:11.599 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13242 ms) 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7741 ms) 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8037 ms) 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13059 ms) 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7909 ms) 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-30T14:03:11.600 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8016 ms) 2024-03-30T14:03:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:11 smithi138 ceph-mon[33397]: pgmap v395: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-03-30T14:03:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:11 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:11 smithi138 ceph-mon[33397]: osdmap e293: 8 total, 8 up, 8 in 2024-03-30T14:03:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:12.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[29392]: pgmap v395: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-03-30T14:03:12.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:12.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[29392]: osdmap e293: 8 total, 8 up, 8 in 2024-03-30T14:03:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[37717]: pgmap v395: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-03-30T14:03:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[37717]: osdmap e293: 8 total, 8 up, 8 in 2024-03-30T14:03:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:12.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:13.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:13 smithi138 ceph-mon[33397]: pgmap v398: 228 pgs: 228 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-30T14:03:13.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:13 smithi138 ceph-mon[33397]: osdmap e294: 8 total, 8 up, 8 in 2024-03-30T14:03:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:13.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:13.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[37717]: pgmap v398: 228 pgs: 228 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-30T14:03:13.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:13.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[37717]: osdmap e294: 8 total, 8 up, 8 in 2024-03-30T14:03:13.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:13.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:13.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[29392]: pgmap v398: 228 pgs: 228 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-30T14:03:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[29392]: osdmap e294: 8 total, 8 up, 8 in 2024-03-30T14:03:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]: dispatch 2024-03-30T14:03:13.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: osdmap e295: 8 total, 8 up, 8 in 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:14.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi102-74825-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:14.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:14.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:14.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: osdmap e295: 8 total, 8 up, 8 in 2024-03-30T14:03:14.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:14.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi102-74825-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2852125764' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi102-74825-55"}]': finished 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: osdmap e295: 8 total, 8 up, 8 in 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:14.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi102-74825-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:15.868 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-mon[33397]: pgmap v401: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi102-74825-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-mon[33397]: osdmap e296: 8 total, 8 up, 8 in 2024-03-30T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi102-74825-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:16.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[37717]: pgmap v401: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:16.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:16.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi102-74825-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[37717]: osdmap e296: 8 total, 8 up, 8 in 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi102-74825-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[29392]: pgmap v401: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi102-74825-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[29392]: osdmap e296: 8 total, 8 up, 8 in 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi102-74825-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:16.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:16.875 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:03:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:17.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: osdmap e297: 8 total, 8 up, 8 in 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi102-74825-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: osdmap e298: 8 total, 8 up, 8 in 2024-03-30T14:03:17.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:03:17.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:17.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:03:17.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: osdmap e297: 8 total, 8 up, 8 in 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi102-74825-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: osdmap e298: 8 total, 8 up, 8 in 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: osdmap e297: 8 total, 8 up, 8 in 2024-03-30T14:03:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:17.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:17.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi102-74825-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:17.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:17.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:03:17.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: osdmap e298: 8 total, 8 up, 8 in 2024-03-30T14:03:17.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:03:18.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:17 smithi138 ceph-mon[33397]: pgmap v404: 260 pgs: 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:18.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:03:18.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:17 smithi138 ceph-mon[33397]: osdmap e299: 8 total, 8 up, 8 in 2024-03-30T14:03:18.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:17 smithi102 ceph-mon[29392]: pgmap v404: 260 pgs: 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:18.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:03:18.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:17 smithi102 ceph-mon[29392]: osdmap e299: 8 total, 8 up, 8 in 2024-03-30T14:03:18.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:17 smithi102 ceph-mon[37717]: pgmap v404: 260 pgs: 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:18.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:03:18.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:17 smithi102 ceph-mon[37717]: osdmap e299: 8 total, 8 up, 8 in 2024-03-30T14:03:19.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:19 smithi138 ceph-mon[33397]: pgmap v407: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:19 smithi138 ceph-mon[33397]: osdmap e300: 8 total, 8 up, 8 in 2024-03-30T14:03:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]: dispatch 2024-03-30T14:03:19.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]: dispatch 2024-03-30T14:03:19.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[29392]: pgmap v407: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:19.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[29392]: osdmap e300: 8 total, 8 up, 8 in 2024-03-30T14:03:19.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:19.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[37717]: pgmap v407: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[37717]: osdmap e300: 8 total, 8 up, 8 in 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]: dispatch 2024-03-30T14:03:19.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]: dispatch 2024-03-30T14:03:20.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:20.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]': finished 2024-03-30T14:03:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:20 smithi138 ceph-mon[33397]: osdmap e301: 8 total, 8 up, 8 in 2024-03-30T14:03:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:20.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:20 smithi138 ceph-mon[33397]: osdmap e302: 8 total, 8 up, 8 in 2024-03-30T14:03:20.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:20.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]': finished 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[37717]: osdmap e301: 8 total, 8 up, 8 in 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[37717]: osdmap e302: 8 total, 8 up, 8 in 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-41"}]': finished 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[29392]: osdmap e301: 8 total, 8 up, 8 in 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]: dispatch 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2538909767' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi102-74825-56"}]': finished 2024-03-30T14:03:20.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:20 smithi102 ceph-mon[29392]: osdmap e302: 8 total, 8 up, 8 in 2024-03-30T14:03:22.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: pgmap v410: 260 pgs: 260 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: osdmap e303: 8 total, 8 up, 8 in 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:22.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: pgmap v410: 260 pgs: 260 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: osdmap e303: 8 total, 8 up, 8 in 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: pgmap v410: 260 pgs: 260 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:22.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:22.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi102-74825-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:22.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: osdmap e303: 8 total, 8 up, 8 in 2024-03-30T14:03:22.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:22.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:22.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:23.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:23 smithi138 ceph-mon[33397]: pgmap v413: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:23.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:23.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi102-75117-6","var": "pg_num","format": "json"}]: dispatch 2024-03-30T14:03:23.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:23.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:23 smithi138 ceph-mon[33397]: osdmap e304: 8 total, 8 up, 8 in 2024-03-30T14:03:23.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:23 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:23.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:23.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[37717]: pgmap v413: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:23.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:23.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi102-75117-6","var": "pg_num","format": "json"}]: dispatch 2024-03-30T14:03:23.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:23.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[37717]: osdmap e304: 8 total, 8 up, 8 in 2024-03-30T14:03:23.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:23.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:23.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[29392]: pgmap v413: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:23.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:23.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi102-75117-6","var": "pg_num","format": "json"}]: dispatch 2024-03-30T14:03:23.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:23.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[29392]: osdmap e304: 8 total, 8 up, 8 in 2024-03-30T14:03:23.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:23.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:24.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: osdmap e305: 8 total, 8 up, 8 in 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]': finished 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: osdmap e306: 8 total, 8 up, 8 in 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:24.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:24.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:24.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:24.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: osdmap e305: 8 total, 8 up, 8 in 2024-03-30T14:03:24.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T14:03:24.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T14:03:24.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]': finished 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: osdmap e306: 8 total, 8 up, 8 in 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi102-74825-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: osdmap e305: 8 total, 8 up, 8 in 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_count","val": "8"}]': finished 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: osdmap e306: 8 total, 8 up, 8 in 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:24.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:25.933 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:03:25.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-mon[33397]: pgmap v416: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:25.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:25.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-mon[33397]: osdmap e307: 8 total, 8 up, 8 in 2024-03-30T14:03:25.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:25.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:25.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T14:03:25.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T14:03:25.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[29392]: pgmap v416: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:25.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:25.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[29392]: osdmap e307: 8 total, 8 up, 8 in 2024-03-30T14:03:25.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:25.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:25.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T14:03:25.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T14:03:25.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[37717]: pgmap v416: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:25.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:25.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[37717]: osdmap e307: 8 total, 8 up, 8 in 2024-03-30T14:03:25.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:25.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:25.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T14:03:25.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T14:03:26.889 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-30T14:03:26.889 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: ok, hit_set contains 262:602f83fe:::foo:head 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9036 ms) 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg_num = 32 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 0 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 1 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 2 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 3 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 4 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 5 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 6 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 7 ls 1711807407,0 2024-03-30T14:03:26.890 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 8 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 9 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 10 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 11 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 12 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 13 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 14 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 15 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 16 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 17 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 18 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 19 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 20 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 21 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 22 ls 1711807407,0 2024-03-30T14:03:26.891 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 23 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 24 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 25 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 26 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 27 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 28 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 29 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 30 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg 31 ls 1711807407,0 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: pg_num = 32 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6cac518f:::0:head 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:02547ec2:::1:head 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f905c69b:::2:head 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:cfc208b3:::3:head 2024-03-30T14:03:26.892 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d83876eb:::4:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b29083e3:::5:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c4fdafeb:::6:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:5c6b0b28:::7:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:bd63b0f1:::8:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:e960b815:::9:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:52ea6a34:::10:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:89d3ae78:::11:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:de5d7c5f:::12:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:566253c9:::13:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:62a1935d:::14:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:863748b0:::15:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:3958e169:::16:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:4d4dabf9:::17:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:8391935d:::18:head 2024-03-30T14:03:26.893 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:28883081:::19:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:69259c59:::20:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:4bdb80b7:::21:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a11c5d71:::22:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:271af37b:::23:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:95b121be:::24:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:58d1031b:::25:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:0a050783:::26:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c709704c:::27:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:cbe56eaf:::28:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:86b4b162:::29:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:70d89383:::30:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:dd450c7c:::31:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6d5729b1:::32:head 2024-03-30T14:03:26.894 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c388f3fb:::33:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:56cfea31:::34:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:9dbc1bf7:::35:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:40b74ccd:::36:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:4d5aaf42:::37:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:920f362c:::38:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6cc53222:::39:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:9cad833f:::40:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:1ea84d41:::41:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c4480ef6:::42:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a694361e:::43:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d1bd33e9:::44:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:ddc2cd5d:::45:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:2b782207:::46:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:7b187fca:::47:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:90ecdf6f:::48:head 2024-03-30T14:03:26.895 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a5ed95fe:::49:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:ea0eaa55:::50:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f33ef17b:::51:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a0d1b2f6:::52:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:60c5229e:::53:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:edcbc575:::54:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:102cf253:::55:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:efb7fb0b:::56:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:50d0a326:::57:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d4dc5daf:::58:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:3a130462:::59:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:ec87ed71:::60:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d5bc9454:::61:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:3ddfe313:::62:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:7c2816b9:::63:head 2024-03-30T14:03:26.896 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:47e00e4d:::64:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c6410c18:::65:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b48ed237:::66:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:cd63ad31:::67:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b179e92b:::68:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:0d9f741a:::69:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6d3352ae:::70:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c6d5c19e:::71:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:bc4729c3:::72:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:77e930b9:::73:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:0abeecfd:::74:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b7c37e15:::75:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b6378398:::76:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:02bd68de:::77:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:cc795d2d:::78:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:630d4fea:::79:head 2024-03-30T14:03:26.897 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:e0d29ef5:::80:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:fd6f13d2:::81:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:606461d5:::82:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:eadbdc43:::83:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:8761d0bb:::84:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:9ef0186f:::85:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:e0d41294:::86:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:961de695:::87:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:1423148f:::88:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:633a8fa2:::89:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a8653809:::90:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:3dac8b33:::91:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:35aad435:::92:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f6dcc343:::93:head 2024-03-30T14:03:26.898 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:dbbdad87:::94:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:1cb48ce0:::95:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:03cd461c:::96:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:17a4ea99:::97:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:9993c9a7:::98:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6394211c:::99:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:94c7ae57:::100:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6fdee5bb:::101:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:9a477fd1:::102:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:eb850916:::103:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:affc56b9:::104:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b42dc814:::105:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f319f8f0:::106:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:9a40b9de:::107:head 2024-03-30T14:03:26.899 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:8b524f28:::108:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:e3de589f:::109:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:90f90a5b:::110:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a7b4f1d7:::111:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:af51766e:::112:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b6f90bd1:::113:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:e0261208:::114:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c9569ef7:::115:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:61bebe50:::116:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:fe93412b:::117:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d3d38bee:::118:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:3100ba0c:::119:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d0560ada:::120:head 2024-03-30T14:03:26.900 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f0ea8b35:::121:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:766f231a:::122:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a07a2582:::123:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:bd7c6b3a:::124:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:fb2ddaff:::125:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:4408e1fe:::126:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:ee1df7a7:::127:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c3002909:::128:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:4f48ffa9:::129:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:edf38733:::130:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c08425c0:::131:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:5f902d98:::132:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:41ea2c93:::133:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:813cee13:::134:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:0131818d:::135:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:26ba5a85:::136:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:381b8a5a:::137:head 2024-03-30T14:03:26.901 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:28797e47:::138:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:bfca7f22:::139:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:36807075:::140:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:80b03975:::141:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:5c15709b:::142:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f39ea15e:::143:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:ea992956:::144:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:48887b1c:::145:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:9f24a9dd:::146:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:987f100b:::147:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d2dd3581:::148:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:7fed1808:::149:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c80b70e9:::150:head 2024-03-30T14:03:26.902 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:85ed90f9:::151:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:36428b24:::152:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d044c34a:::153:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:7c18bf58:::154:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d1c21232:::155:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a7a3c575:::156:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:87da0633:::157:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d5ac3822:::158:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:3f20522d:::159:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6ca26563:::160:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:532ce135:::161:head 2024-03-30T14:03:26.903 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:c78863e6:::162:head 2024-03-30T14:03:26.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:26.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:26.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-30T14:03:26.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:26 smithi138 ceph-mon[33397]: osdmap e308: 8 total, 8 up, 8 in 2024-03-30T14:03:26.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:26.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:26.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:03:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:26.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:26.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:26.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-30T14:03:26.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[37717]: osdmap e308: 8 total, 8 up, 8 in 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[29392]: osdmap e308: 8 total, 8 up, 8 in 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3803054407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:26.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]: dispatch 2024-03-30T14:03:27.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: pgmap v419: 260 pgs: 260 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:27.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi102-75117-43","var": "pg_num","format": "json"}]: dispatch 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]: dispatch 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]: dispatch 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]': finished 2024-03-30T14:03:27.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:27 smithi138 ceph-mon[33397]: osdmap e309: 8 total, 8 up, 8 in 2024-03-30T14:03:27.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: pgmap v419: 260 pgs: 260 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:27.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:27.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi102-75117-43","var": "pg_num","format": "json"}]: dispatch 2024-03-30T14:03:27.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:27.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:27.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]: dispatch 2024-03-30T14:03:27.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]: dispatch 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]': finished 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[29392]: osdmap e309: 8 total, 8 up, 8 in 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: pgmap v419: 260 pgs: 260 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi102-75117-43","var": "pg_num","format": "json"}]: dispatch 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]: dispatch 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]: dispatch 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi102-74825-57"}]': finished 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-43"}]': finished 2024-03-30T14:03:27.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:27 smithi102 ceph-mon[37717]: osdmap e309: 8 total, 8 up, 8 in 2024-03-30T14:03:28.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:28.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:28.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:28.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:28.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:30.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:29 smithi138 ceph-mon[33397]: pgmap v422: 260 pgs: 260 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:30.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:30.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:29 smithi138 ceph-mon[33397]: osdmap e310: 8 total, 8 up, 8 in 2024-03-30T14:03:30.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:30.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:30.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[29392]: pgmap v422: 260 pgs: 260 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:30.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:30.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[29392]: osdmap e310: 8 total, 8 up, 8 in 2024-03-30T14:03:30.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:30.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:30.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[37717]: pgmap v422: 260 pgs: 260 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:30.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi102-74825-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:30.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[37717]: osdmap e310: 8 total, 8 up, 8 in 2024-03-30T14:03:30.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:30.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:31.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: osdmap e311: 8 total, 8 up, 8 in 2024-03-30T14:03:31.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:31.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:31.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:31.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:31.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: osdmap e312: 8 total, 8 up, 8 in 2024-03-30T14:03:31.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:31.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: osdmap e311: 8 total, 8 up, 8 in 2024-03-30T14:03:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: osdmap e312: 8 total, 8 up, 8 in 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: osdmap e311: 8 total, 8 up, 8 in 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi102-74825-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: osdmap e312: 8 total, 8 up, 8 in 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:32.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:31 smithi138 ceph-mon[33397]: pgmap v425: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:32.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:31 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:32.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:32.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:31 smithi138 ceph-mon[33397]: osdmap e313: 8 total, 8 up, 8 in 2024-03-30T14:03:32.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:03:32.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:03:32.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[29392]: pgmap v425: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:32.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:32.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:32.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[29392]: osdmap e313: 8 total, 8 up, 8 in 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[37717]: pgmap v425: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[37717]: osdmap e313: 8 total, 8 up, 8 in 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:03:32.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:03:34.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:33 smithi138 ceph-mon[33397]: pgmap v428: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]': finished 2024-03-30T14:03:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:33 smithi138 ceph-mon[33397]: osdmap e314: 8 total, 8 up, 8 in 2024-03-30T14:03:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:03:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:03:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:34.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[29392]: pgmap v428: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:34.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]': finished 2024-03-30T14:03:34.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[29392]: osdmap e314: 8 total, 8 up, 8 in 2024-03-30T14:03:34.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:03:34.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[37717]: pgmap v428: 268 pgs: 8 unknown, 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_count","val": "3"}]': finished 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[37717]: osdmap e314: 8 total, 8 up, 8 in 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:34.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:35.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]': finished 2024-03-30T14:03:35.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: osdmap e315: 8 total, 8 up, 8 in 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: osdmap e316: 8 total, 8 up, 8 in 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:03:35.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:03:35.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]': finished 2024-03-30T14:03:35.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:35.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: osdmap e315: 8 total, 8 up, 8 in 2024-03-30T14:03:35.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:35.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:35.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: osdmap e316: 8 total, 8 up, 8 in 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_period","val": "3"}]': finished 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: osdmap e315: 8 total, 8 up, 8 in 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:35.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2179333337' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]: dispatch 2024-03-30T14:03:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:03:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi102-74825-58"}]': finished 2024-03-30T14:03:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: osdmap e316: 8 total, 8 up, 8 in 2024-03-30T14:03:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:03:35.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:03:35.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: pgmap v431: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:35.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: osdmap e317: 8 total, 8 up, 8 in 2024-03-30T14:03:35.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:35.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: pgmap v431: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:36.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: osdmap e317: 8 total, 8 up, 8 in 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: pgmap v431: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:36.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: osdmap e317: 8 total, 8 up, 8 in 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:36.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:03:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:03:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:37.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:37.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:36 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:37.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:37.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:36 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:36 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:38.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:37 smithi138 ceph-mon[33397]: pgmap v434: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:38.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:37 smithi138 ceph-mon[33397]: osdmap e318: 8 total, 8 up, 8 in 2024-03-30T14:03:38.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:38.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:37 smithi138 ceph-mon[33397]: osdmap e319: 8 total, 8 up, 8 in 2024-03-30T14:03:38.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[29392]: pgmap v434: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:38.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[29392]: osdmap e318: 8 total, 8 up, 8 in 2024-03-30T14:03:38.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:38.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[29392]: osdmap e319: 8 total, 8 up, 8 in 2024-03-30T14:03:38.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[37717]: pgmap v434: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:38.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[37717]: osdmap e318: 8 total, 8 up, 8 in 2024-03-30T14:03:38.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi102-74825-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:38.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:37 smithi102 ceph-mon[37717]: osdmap e319: 8 total, 8 up, 8 in 2024-03-30T14:03:40.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:39 smithi138 ceph-mon[33397]: pgmap v437: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:40.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:39 smithi138 ceph-mon[33397]: osdmap e320: 8 total, 8 up, 8 in 2024-03-30T14:03:40.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:39 smithi102 ceph-mon[29392]: pgmap v437: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:40.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:39 smithi102 ceph-mon[29392]: osdmap e320: 8 total, 8 up, 8 in 2024-03-30T14:03:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:39 smithi102 ceph-mon[37717]: pgmap v437: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:39 smithi102 ceph-mon[37717]: osdmap e320: 8 total, 8 up, 8 in 2024-03-30T14:03:41.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:40 smithi138 ceph-mon[33397]: osdmap e321: 8 total, 8 up, 8 in 2024-03-30T14:03:41.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:41.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:41.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:40 smithi102 ceph-mon[29392]: osdmap e321: 8 total, 8 up, 8 in 2024-03-30T14:03:41.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:41.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:41.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:40 smithi102 ceph-mon[37717]: osdmap e321: 8 total, 8 up, 8 in 2024-03-30T14:03:41.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:41.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:7462ddf6:::1o.RoundTripAppendPP (3000 ms) 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2999 ms) 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3000 ms) 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3007 ms) 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: setting pool EIO 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: max_success 100, min_failed 101 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4027 ms) 2024-03-30T14:03:41.710 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3022 ms) 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (131377 ms total) 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3259 ms) 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8776 ms) 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14914 ms) 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3009 ms) 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29958 ms total) 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-30T14:03:41.711 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2999 ms) 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2999 ms total) 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13995 ms) 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7023 ms) 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6986 ms) 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7032 ms) 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3015 ms) 2024-03-30T14:03:41.712 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7018 ms) 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7034 ms) 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6910 ms) 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7019 ms) 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7018 ms) 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7024 ms) 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7037 ms) 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-30T14:03:41.713 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7020 ms) 2024-03-30T14:03:42.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:41 smithi138 ceph-mon[33397]: pgmap v440: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:03:42.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:42.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:41 smithi138 ceph-mon[33397]: osdmap e322: 8 total, 8 up, 8 in 2024-03-30T14:03:42.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:42.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:42.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[29392]: pgmap v440: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:03:42.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:42.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[29392]: osdmap e322: 8 total, 8 up, 8 in 2024-03-30T14:03:42.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:42.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:42.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[37717]: pgmap v440: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:03:42.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:42.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[37717]: osdmap e322: 8 total, 8 up, 8 in 2024-03-30T14:03:42.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1181812560' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:42.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]: dispatch 2024-03-30T14:03:43.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:43.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:42 smithi138 ceph-mon[33397]: osdmap e323: 8 total, 8 up, 8 in 2024-03-30T14:03:43.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:43.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:43.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:42 smithi102 ceph-mon[29392]: osdmap e323: 8 total, 8 up, 8 in 2024-03-30T14:03:43.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi102-74825-59"}]': finished 2024-03-30T14:03:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:42 smithi102 ceph-mon[37717]: osdmap e323: 8 total, 8 up, 8 in 2024-03-30T14:03:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:43 smithi138 ceph-mon[33397]: pgmap v443: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:03:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:43 smithi138 ceph-mon[33397]: osdmap e324: 8 total, 8 up, 8 in 2024-03-30T14:03:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1874838906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[29392]: pgmap v443: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:03:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[29392]: osdmap e324: 8 total, 8 up, 8 in 2024-03-30T14:03:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1874838906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:44.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[37717]: pgmap v443: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:03:44.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[37717]: osdmap e324: 8 total, 8 up, 8 in 2024-03-30T14:03:44.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1874838906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:44.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:44 smithi138 ceph-mon[33397]: osdmap e325: 8 total, 8 up, 8 in 2024-03-30T14:03:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:44 smithi102 ceph-mon[37717]: osdmap e325: 8 total, 8 up, 8 in 2024-03-30T14:03:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi102-74825-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:44 smithi102 ceph-mon[29392]: osdmap e325: 8 total, 8 up, 8 in 2024-03-30T14:03:45.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: pgmap v446: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: osdmap e326: 8 total, 8 up, 8 in 2024-03-30T14:03:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:45.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:46.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: pgmap v446: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: osdmap e326: 8 total, 8 up, 8 in 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: pgmap v446: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: osdmap e326: 8 total, 8 up, 8 in 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:46.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:46.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:46.760 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:03:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:47.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:47.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:47.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:46 smithi138 ceph-mon[33397]: osdmap e327: 8 total, 8 up, 8 in 2024-03-30T14:03:47.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:47.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[37717]: osdmap e327: 8 total, 8 up, 8 in 2024-03-30T14:03:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:47.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:47.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi102-74825-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:47.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[29392]: osdmap e327: 8 total, 8 up, 8 in 2024-03-30T14:03:47.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:47.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:47 smithi138 ceph-mon[33397]: pgmap v449: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:03:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:47 smithi138 ceph-mon[33397]: osdmap e328: 8 total, 8 up, 8 in 2024-03-30T14:03:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:47 smithi102 ceph-mon[37717]: pgmap v449: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:03:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:47 smithi102 ceph-mon[37717]: osdmap e328: 8 total, 8 up, 8 in 2024-03-30T14:03:48.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:47 smithi102 ceph-mon[29392]: pgmap v449: 260 pgs: 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:03:48.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:47 smithi102 ceph-mon[29392]: osdmap e328: 8 total, 8 up, 8 in 2024-03-30T14:03:49.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:49.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:48 smithi138 ceph-mon[33397]: osdmap e329: 8 total, 8 up, 8 in 2024-03-30T14:03:49.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:49.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:49.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]: dispatch 2024-03-30T14:03:49.083 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]: dispatch 2024-03-30T14:03:49.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:49.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[29392]: osdmap e329: 8 total, 8 up, 8 in 2024-03-30T14:03:49.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:49.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:49.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]: dispatch 2024-03-30T14:03:49.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]: dispatch 2024-03-30T14:03:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi102-74825-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[37717]: osdmap e329: 8 total, 8 up, 8 in 2024-03-30T14:03:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:03:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]: dispatch 2024-03-30T14:03:49.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]: dispatch 2024-03-30T14:03:50.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:49 smithi138 ceph-mon[33397]: pgmap v452: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:03:50.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:49 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:50.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]': finished 2024-03-30T14:03:50.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:49 smithi138 ceph-mon[33397]: osdmap e330: 8 total, 8 up, 8 in 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[37717]: pgmap v452: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]': finished 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[37717]: osdmap e330: 8 total, 8 up, 8 in 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[29392]: pgmap v452: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-45"}]': finished 2024-03-30T14:03:50.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:49 smithi102 ceph-mon[29392]: osdmap e330: 8 total, 8 up, 8 in 2024-03-30T14:03:51.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:50 smithi138 ceph-mon[33397]: osdmap e331: 8 total, 8 up, 8 in 2024-03-30T14:03:51.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:51.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:51.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:50 smithi102 ceph-mon[29392]: osdmap e331: 8 total, 8 up, 8 in 2024-03-30T14:03:51.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:51.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:51.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:50 smithi102 ceph-mon[37717]: osdmap e331: 8 total, 8 up, 8 in 2024-03-30T14:03:51.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:51.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:52.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:51 smithi138 ceph-mon[33397]: pgmap v455: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:52.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:52.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:51 smithi138 ceph-mon[33397]: osdmap e332: 8 total, 8 up, 8 in 2024-03-30T14:03:52.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:52.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:52.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:52.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[37717]: pgmap v455: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[37717]: osdmap e332: 8 total, 8 up, 8 in 2024-03-30T14:03:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[29392]: pgmap v455: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[29392]: osdmap e332: 8 total, 8 up, 8 in 2024-03-30T14:03:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1763712776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]: dispatch 2024-03-30T14:03:52.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:03:53.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:53.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: osdmap e333: 8 total, 8 up, 8 in 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:53.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:53.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:53.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:53.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: osdmap e333: 8 total, 8 up, 8 in 2024-03-30T14:03:53.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi102-74825-61"}]': finished 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: osdmap e333: 8 total, 8 up, 8 in 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:53.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:53.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:03:53.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:53.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:03:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: pgmap v458: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:03:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: osdmap e334: 8 total, 8 up, 8 in 2024-03-30T14:03:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: pgmap v458: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: osdmap e334: 8 total, 8 up, 8 in 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: pgmap v458: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: osdmap e334: 8 total, 8 up, 8 in 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:03:55.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]': finished 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]: dispatch 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: osdmap e335: 8 total, 8 up, 8 in 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]: dispatch 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]': finished 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: osdmap e336: 8 total, 8 up, 8 in 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:55.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]': finished 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]: dispatch 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: osdmap e335: 8 total, 8 up, 8 in 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]: dispatch 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]': finished 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: osdmap e336: 8 total, 8 up, 8 in 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:55.339 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:55.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-47"}]': finished 2024-03-30T14:03:55.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]: dispatch 2024-03-30T14:03:55.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: osdmap e335: 8 total, 8 up, 8 in 2024-03-30T14:03:55.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]: dispatch 2024-03-30T14:03:55.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:03:55.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi102-74825-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:03:55.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-47", "mode": "writeback"}]': finished 2024-03-30T14:03:55.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: osdmap e336: 8 total, 8 up, 8 in 2024-03-30T14:03:55.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:55.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:03:55.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:03:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:03:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:56 smithi138 ceph-mon[33397]: pgmap v461: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:03:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:56 smithi138 ceph-mon[33397]: osdmap e337: 8 total, 8 up, 8 in 2024-03-30T14:03:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[37717]: pgmap v461: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[37717]: osdmap e337: 8 total, 8 up, 8 in 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[29392]: pgmap v461: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[29392]: osdmap e337: 8 total, 8 up, 8 in 2024-03-30T14:03:56.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:56.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:03:56.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:03:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:03:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:03:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: osdmap e338: 8 total, 8 up, 8 in 2024-03-30T14:03:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: osdmap e338: 8 total, 8 up, 8 in 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: osdmap e338: 8 total, 8 up, 8 in 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:03:57.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:03:58.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: pgmap v464: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:03:58.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:03:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:03:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: osdmap e339: 8 total, 8 up, 8 in 2024-03-30T14:03:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:03:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:03:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: pgmap v464: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: osdmap e339: 8 total, 8 up, 8 in 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:03:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: pgmap v464: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: osdmap e339: 8 total, 8 up, 8 in 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1368832124' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:03:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]: dispatch 2024-03-30T14:04:00.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: pgmap v467: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: osdmap e340: 8 total, 8 up, 8 in 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:00.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:03:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:00.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: pgmap v467: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:04:00.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:04:00.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:04:00.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: osdmap e340: 8 total, 8 up, 8 in 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: pgmap v467: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:04:00.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi102-74825-62"}]': finished 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: osdmap e340: 8 total, 8 up, 8 in 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:03:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:01.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T14:04:01.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:01.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:04:01.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:01.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:00 smithi138 ceph-mon[33397]: osdmap e341: 8 total, 8 up, 8 in 2024-03-30T14:04:01.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:04:01.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[37717]: osdmap e341: 8 total, 8 up, 8 in 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:01.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T14:04:01.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi102-74825-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:01.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:04:01.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:01.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[29392]: osdmap e341: 8 total, 8 up, 8 in 2024-03-30T14:04:01.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:04:01.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:02.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:01 smithi138 ceph-mon[33397]: pgmap v470: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:02.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T14:04:02.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:02.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:01 smithi138 ceph-mon[33397]: osdmap e342: 8 total, 8 up, 8 in 2024-03-30T14:04:02.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:02.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[37717]: pgmap v470: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:02.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T14:04:02.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:02.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[37717]: osdmap e342: 8 total, 8 up, 8 in 2024-03-30T14:04:02.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:02.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[29392]: pgmap v470: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T14:04:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[29392]: osdmap e342: 8 total, 8 up, 8 in 2024-03-30T14:04:02.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:03.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:03.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:03.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:02 smithi138 ceph-mon[33397]: osdmap e343: 8 total, 8 up, 8 in 2024-03-30T14:04:03.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:03.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:03.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:03.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:03.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[37717]: osdmap e343: 8 total, 8 up, 8 in 2024-03-30T14:04:03.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:03.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:03.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi102-74825-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:03.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:03.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[29392]: osdmap e343: 8 total, 8 up, 8 in 2024-03-30T14:04:03.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:03.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:04.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: pgmap v473: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:04.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]': finished 2024-03-30T14:04:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: osdmap e344: 8 total, 8 up, 8 in 2024-03-30T14:04:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:04.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: pgmap v473: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]': finished 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: osdmap e344: 8 total, 8 up, 8 in 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: pgmap v473: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]': finished 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: osdmap e344: 8 total, 8 up, 8 in 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:04.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-47"}]: dispatch 2024-03-30T14:04:05.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:04 smithi138 ceph-mon[33397]: osdmap e345: 8 total, 8 up, 8 in 2024-03-30T14:04:05.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:05.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:04 smithi102 ceph-mon[37717]: osdmap e345: 8 total, 8 up, 8 in 2024-03-30T14:04:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:05.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:04 smithi102 ceph-mon[29392]: osdmap e345: 8 total, 8 up, 8 in 2024-03-30T14:04:05.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:05.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:05.877 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:04:06.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-mon[33397]: pgmap v476: 228 pgs: 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:06.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:06.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-mon[33397]: osdmap e346: 8 total, 8 up, 8 in 2024-03-30T14:04:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:06.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[29392]: pgmap v476: 228 pgs: 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[29392]: osdmap e346: 8 total, 8 up, 8 in 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:06.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[37717]: pgmap v476: 228 pgs: 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[37717]: osdmap e346: 8 total, 8 up, 8 in 2024-03-30T14:04:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1943893772' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]: dispatch 2024-03-30T14:04:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:06.882 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:04:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: osdmap e347: 8 total, 8 up, 8 in 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi102-74825-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:04:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi102-74825-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: osdmap e348: 8 total, 8 up, 8 in 2024-03-30T14:04:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi102-74825-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:07.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: osdmap e347: 8 total, 8 up, 8 in 2024-03-30T14:04:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi102-74825-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi102-74825-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: osdmap e348: 8 total, 8 up, 8 in 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi102-74825-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:07.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi102-74825-63"}]': finished 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: osdmap e347: 8 total, 8 up, 8 in 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi102-74825-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi102-74825-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: osdmap e348: 8 total, 8 up, 8 in 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi102-74825-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:07.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:08.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:08 smithi138 ceph-mon[33397]: pgmap v479: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:04:08.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:08 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:04:08.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]': finished 2024-03-30T14:04:08.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:08 smithi102 ceph-mon[29392]: pgmap v479: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:04:08.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:08 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:04:08.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]': finished 2024-03-30T14:04:08.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:08 smithi102 ceph-mon[37717]: pgmap v479: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:04:08.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:08 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:04:08.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-49"}]': finished 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]: dispatch 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: osdmap e349: 8 total, 8 up, 8 in 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]: dispatch 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi102-74825-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]': finished 2024-03-30T14:04:09.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:09 smithi138 ceph-mon[33397]: osdmap e350: 8 total, 8 up, 8 in 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]: dispatch 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: osdmap e349: 8 total, 8 up, 8 in 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]: dispatch 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi102-74825-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]': finished 2024-03-30T14:04:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[29392]: osdmap e350: 8 total, 8 up, 8 in 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]: dispatch 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: osdmap e349: 8 total, 8 up, 8 in 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]: dispatch 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi102-74825-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-49", "mode": "readproxy"}]': finished 2024-03-30T14:04:09.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:09 smithi102 ceph-mon[37717]: osdmap e350: 8 total, 8 up, 8 in 2024-03-30T14:04:10.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:10 smithi138 ceph-mon[33397]: pgmap v482: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:04:10.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:10 smithi138 ceph-mon[33397]: osdmap e351: 8 total, 8 up, 8 in 2024-03-30T14:04:10.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:10 smithi102 ceph-mon[29392]: pgmap v482: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:04:10.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:10 smithi102 ceph-mon[29392]: osdmap e351: 8 total, 8 up, 8 in 2024-03-30T14:04:10.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:10 smithi102 ceph-mon[37717]: pgmap v482: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:04:10.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:10 smithi102 ceph-mon[37717]: osdmap e351: 8 total, 8 up, 8 in 2024-03-30T14:04:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:11 smithi138 ceph-mon[33397]: pgmap v485: 268 pgs: 268 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:04:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:11 smithi138 ceph-mon[33397]: osdmap e352: 8 total, 8 up, 8 in 2024-03-30T14:04:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:11 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:12.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[37717]: pgmap v485: 268 pgs: 268 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[37717]: osdmap e352: 8 total, 8 up, 8 in 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[29392]: pgmap v485: 268 pgs: 268 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[29392]: osdmap e352: 8 total, 8 up, 8 in 2024-03-30T14:04:12.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:12.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:13.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:13.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:12 smithi138 ceph-mon[33397]: osdmap e353: 8 total, 8 up, 8 in 2024-03-30T14:04:13.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:13.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:13.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:13.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[29392]: osdmap e353: 8 total, 8 up, 8 in 2024-03-30T14:04:13.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:13.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:13.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:13.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[37717]: osdmap e353: 8 total, 8 up, 8 in 2024-03-30T14:04:13.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]: dispatch 2024-03-30T14:04:13.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/669171503' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi102-74825-64"}]': finished 2024-03-30T14:04:14.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: pgmap v488: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:04:14.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: osdmap e354: 8 total, 8 up, 8 in 2024-03-30T14:04:14.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:14.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: pgmap v488: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:04:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: osdmap e354: 8 total, 8 up, 8 in 2024-03-30T14:04:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: pgmap v488: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: osdmap e354: 8 total, 8 up, 8 in 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:15.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:15.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:14 smithi138 ceph-mon[33397]: osdmap e355: 8 total, 8 up, 8 in 2024-03-30T14:04:15.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:15.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:15.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:15.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[29392]: osdmap e355: 8 total, 8 up, 8 in 2024-03-30T14:04:15.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:15.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:15.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi102-74825-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:15.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[37717]: osdmap e355: 8 total, 8 up, 8 in 2024-03-30T14:04:15.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:15.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:15.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:04:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:04:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:15 smithi138 ceph-mon[33397]: pgmap v491: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:15 smithi138 ceph-mon[33397]: osdmap e356: 8 total, 8 up, 8 in 2024-03-30T14:04:16.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:15 smithi102 ceph-mon[29392]: pgmap v491: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:16.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:15 smithi102 ceph-mon[29392]: osdmap e356: 8 total, 8 up, 8 in 2024-03-30T14:04:16.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:15 smithi102 ceph-mon[37717]: pgmap v491: 260 pgs: 260 active+clean; 8.4 MiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:16.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:15 smithi102 ceph-mon[37717]: osdmap e356: 8 total, 8 up, 8 in 2024-03-30T14:04:16.918 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:04:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:17.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:17.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:17.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:16 smithi138 ceph-mon[33397]: osdmap e357: 8 total, 8 up, 8 in 2024-03-30T14:04:17.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:16 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[37717]: osdmap e357: 8 total, 8 up, 8 in 2024-03-30T14:04:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:17.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi102-74825-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[29392]: osdmap e357: 8 total, 8 up, 8 in 2024-03-30T14:04:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:16 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:18.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:17 smithi138 ceph-mon[33397]: pgmap v494: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:18.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:17 smithi138 ceph-mon[33397]: osdmap e358: 8 total, 8 up, 8 in 2024-03-30T14:04:18.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:17 smithi102 ceph-mon[29392]: pgmap v494: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:18.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:17 smithi102 ceph-mon[29392]: osdmap e358: 8 total, 8 up, 8 in 2024-03-30T14:04:18.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:17 smithi102 ceph-mon[37717]: pgmap v494: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:18.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:17 smithi102 ceph-mon[37717]: osdmap e358: 8 total, 8 up, 8 in 2024-03-30T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:18 smithi138 ceph-mon[33397]: osdmap e359: 8 total, 8 up, 8 in 2024-03-30T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:18 smithi102 ceph-mon[37717]: osdmap e359: 8 total, 8 up, 8 in 2024-03-30T14:04:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:19.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:19.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:18 smithi102 ceph-mon[29392]: osdmap e359: 8 total, 8 up, 8 in 2024-03-30T14:04:19.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:19.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:19 smithi138 ceph-mon[33397]: pgmap v497: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:19 smithi138 ceph-mon[33397]: osdmap e360: 8 total, 8 up, 8 in 2024-03-30T14:04:20.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:20.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:20.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:20.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[29392]: pgmap v497: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[29392]: osdmap e360: 8 total, 8 up, 8 in 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[37717]: pgmap v497: 260 pgs: 260 active+clean; 8.4 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:20.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:20.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[37717]: osdmap e360: 8 total, 8 up, 8 in 2024-03-30T14:04:20.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1334088594' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:20.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]: dispatch 2024-03-30T14:04:20.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:20.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:21.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:21.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:21.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: osdmap e361: 8 total, 8 up, 8 in 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:04:21.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]': finished 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: osdmap e361: 8 total, 8 up, 8 in 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]': finished 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi102-74825-65"}]': finished 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: osdmap e361: 8 total, 8 up, 8 in 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:04:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:04:21.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]': finished 2024-03-30T14:04:22.197 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP63:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:5d165639:::164:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f43765fc:::165:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b4c720e9:::166:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:e694b040:::167:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:afa38db2:::168:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:77ba9f53:::169:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:87495034:::170:head 2024-03-30T14:04:22.218 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:7c96bf0e:::171:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:dbe346cc:::172:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:e943ec24:::173:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f97a9c0c:::174:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:6f26e74d:::175:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:4f95e106:::176:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:0e6f2f8f:::177:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:05db05f1:::178:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:38a78d66:::179:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:d095610b:::180:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:a1a9d709:::181:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:1e5d39db:::182:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:f7df4fb9:::183:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:03a7f161:::184:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:ba70721e:::185:head 2024-03-30T14:04:22.219 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:28e5662d:::186:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:973d52de:::187:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:4303eb1c:::188:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:b990b48e:::189:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:29b8165b:::190:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:3547f197:::191:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:7e260936:::192:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:1abec7b1:::193:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:10fdda93:::194:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:15817eea:::195:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:770bab57:::196:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:ed9e13e7:::197:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:71471a8f:::198:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: checking for 264:10fb1d02:::199:head 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8050 ms) 2024-03-30T14:04:22.220 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: first is 1711807416 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,1711807421,1711807422,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,1711807421,1711807422,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,1711807421,1711807422,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,1711807421,1711807422,1711807424,1711807425,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,1711807421,1711807422,1711807424,1711807425,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807416,1711807418,1711807419,1711807421,1711807422,1711807424,1711807425,0 2024-03-30T14:04:22.221 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807419,1711807421,1711807422,1711807424,1711807425,1711807427,1711807428,0 2024-03-30T14:04:22.222 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: first now 1711807419, trimmed 2024-03-30T14:04:22.222 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21106 ms) 2024-03-30T14:04:22.222 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-30T14:04:22.222 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: foo0 2024-03-30T14:04:22.222 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-30T14:04:22.222 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14072 ms) 2024-03-30T14:04:22.222 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-30T14:04:22.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:22.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:22.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: osdmap e362: 8 total, 8 up, 8 in 2024-03-30T14:04:22.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:22.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:22.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:22.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:22.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:22.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: osdmap e362: 8 total, 8 up, 8 in 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:22.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi102-74825-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: osdmap e362: 8 total, 8 up, 8 in 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-49"}]: dispatch 2024-03-30T14:04:23.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:23 smithi138 ceph-mon[33397]: osdmap e363: 8 total, 8 up, 8 in 2024-03-30T14:04:23.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:23.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:23 smithi138 ceph-mon[33397]: osdmap e364: 8 total, 8 up, 8 in 2024-03-30T14:04:23.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:23.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[29392]: osdmap e363: 8 total, 8 up, 8 in 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[29392]: osdmap e364: 8 total, 8 up, 8 in 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[37717]: osdmap e363: 8 total, 8 up, 8 in 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi102-74825-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[37717]: osdmap e364: 8 total, 8 up, 8 in 2024-03-30T14:04:23.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:23.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:24.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:24 smithi138 ceph-mon[33397]: pgmap v503: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:24.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:24.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:24 smithi138 ceph-mon[33397]: osdmap e365: 8 total, 8 up, 8 in 2024-03-30T14:04:24.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:24.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:24.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[29392]: pgmap v503: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:24.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:24.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[29392]: osdmap e365: 8 total, 8 up, 8 in 2024-03-30T14:04:24.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:24.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:24.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[37717]: pgmap v503: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:04:24.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:24.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[37717]: osdmap e365: 8 total, 8 up, 8 in 2024-03-30T14:04:24.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:24.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:04:25.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: pgmap v506: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: osdmap e366: 8 total, 8 up, 8 in 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:25 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: pgmap v506: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: osdmap e366: 8 total, 8 up, 8 in 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: pgmap v506: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: osdmap e366: 8 total, 8 up, 8 in 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:25 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:26.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:04:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ 2 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3037 ms) 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7066 ms) 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7016 ms) 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7039 ms) 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7019 ms) 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7057 ms) 2024-03-30T14:04:26.970 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-30T14:04:26.971 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7022 ms) 2024-03-30T14:04:26.971 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139388 ms total) 2024-03-30T14:04:26.971 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: 2024-03-30T14:04:26.971 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-30T14:04:26.971 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (303722 ms total) 2024-03-30T14:04:26.971 INFO:tasks.workunit.client.0.smithi102.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-30T14:04:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]': finished 2024-03-30T14:04:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: osdmap e367: 8 total, 8 up, 8 in 2024-03-30T14:04:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]: dispatch 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]: dispatch 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]': finished 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:27.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:26 smithi138 ceph-mon[33397]: osdmap e368: 8 total, 8 up, 8 in 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]': finished 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: osdmap e367: 8 total, 8 up, 8 in 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]: dispatch 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]: dispatch 2024-03-30T14:04:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]': finished 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[29392]: osdmap e368: 8 total, 8 up, 8 in 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-51"}]': finished 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: osdmap e367: 8 total, 8 up, 8 in 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]: dispatch 2024-03-30T14:04:27.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1756260223' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]: dispatch 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]: dispatch 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-51", "mode": "writeback"}]': finished 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi102-74825-66"}]': finished 2024-03-30T14:04:27.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:26 smithi102 ceph-mon[37717]: osdmap e368: 8 total, 8 up, 8 in 2024-03-30T14:04:28.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:27 smithi138 ceph-mon[33397]: pgmap v509: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:04:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:04:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:04:28.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:27 smithi102 ceph-mon[29392]: pgmap v509: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:04:28.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:04:28.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:04:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:27 smithi102 ceph-mon[37717]: pgmap v509: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:04:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:04:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:04:29.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:04:29.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:04:29.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:28 smithi138 ceph-mon[33397]: osdmap e369: 8 total, 8 up, 8 in 2024-03-30T14:04:29.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[29392]: osdmap e369: 8 total, 8 up, 8 in 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[37717]: osdmap e369: 8 total, 8 up, 8 in 2024-03-30T14:04:29.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:04:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:29 smithi138 ceph-mon[33397]: pgmap v512: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:04:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:04:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:04:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:29 smithi138 ceph-mon[33397]: osdmap e370: 8 total, 8 up, 8 in 2024-03-30T14:04:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[29392]: pgmap v512: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[29392]: osdmap e370: 8 total, 8 up, 8 in 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[37717]: pgmap v512: 260 pgs: 260 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:04:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[37717]: osdmap e370: 8 total, 8 up, 8 in 2024-03-30T14:04:30.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:04:31.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:30 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:04:31.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:04:31.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:30 smithi138 ceph-mon[33397]: osdmap e371: 8 total, 8 up, 8 in 2024-03-30T14:04:31.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:04:31.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:04:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:04:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:04:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[29392]: osdmap e371: 8 total, 8 up, 8 in 2024-03-30T14:04:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:04:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:04:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:04:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:04:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[37717]: osdmap e371: 8 total, 8 up, 8 in 2024-03-30T14:04:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:04:31.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:04:32.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:32 smithi138 ceph-mon[33397]: pgmap v515: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:32.483 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:04:32.483 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:04:32.483 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:32 smithi138 ceph-mon[33397]: osdmap e372: 8 total, 8 up, 8 in 2024-03-30T14:04:32.484 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:04:32.484 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:32 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:32.484 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[29392]: pgmap v515: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:32.484 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:04:32.484 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:04:32.484 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[29392]: osdmap e372: 8 total, 8 up, 8 in 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[37717]: pgmap v515: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[37717]: osdmap e372: 8 total, 8 up, 8 in 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:04:32.485 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:32 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:33.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]': finished 2024-03-30T14:04:33.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:33 smithi138 ceph-mon[33397]: osdmap e373: 8 total, 8 up, 8 in 2024-03-30T14:04:33.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]': finished 2024-03-30T14:04:33.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:33 smithi102 ceph-mon[29392]: osdmap e373: 8 total, 8 up, 8 in 2024-03-30T14:04:33.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-51","var": "target_max_objects","val": "1"}]': finished 2024-03-30T14:04:33.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:33 smithi102 ceph-mon[37717]: osdmap e373: 8 total, 8 up, 8 in 2024-03-30T14:04:34.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:34 smithi138 ceph-mon[33397]: pgmap v518: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:34.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:34 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T14:04:34.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:34 smithi102 ceph-mon[29392]: pgmap v518: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:34.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:34 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T14:04:34.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:34 smithi102 ceph-mon[37717]: pgmap v518: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:34.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:34 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T14:04:35.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:04:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:04:36.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:36 smithi138 ceph-mon[33397]: pgmap v519: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:04:36.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:36 smithi102 ceph-mon[29392]: pgmap v519: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:04:36.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:36 smithi102 ceph-mon[37717]: pgmap v519: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:04:36.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:04:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:04:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:37.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:38.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:38 smithi138 ceph-mon[33397]: pgmap v520: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:38.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:38 smithi102 ceph-mon[29392]: pgmap v520: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:38.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:38 smithi102 ceph-mon[37717]: pgmap v520: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-30T14:04:40.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:40 smithi138 ceph-mon[33397]: pgmap v521: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-30T14:04:40.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:40 smithi102 ceph-mon[29392]: pgmap v521: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-30T14:04:40.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:40 smithi102 ceph-mon[37717]: pgmap v521: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-30T14:04:42.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:42 smithi138 ceph-mon[33397]: pgmap v522: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:04:42.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:42 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:42.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:42.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:42.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[29392]: pgmap v522: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[37717]: pgmap v522: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:42.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:42.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:43 smithi138 ceph-mon[33397]: pgmap v523: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:04:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:43 smithi138 ceph-mon[33397]: osdmap e374: 8 total, 8 up, 8 in 2024-03-30T14:04:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[37717]: pgmap v523: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[37717]: osdmap e374: 8 total, 8 up, 8 in 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[29392]: pgmap v523: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[29392]: osdmap e374: 8 total, 8 up, 8 in 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]': finished 2024-03-30T14:04:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:44 smithi138 ceph-mon[33397]: osdmap e375: 8 total, 8 up, 8 in 2024-03-30T14:04:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:45.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]': finished 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[29392]: osdmap e375: 8 total, 8 up, 8 in 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]': finished 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[37717]: osdmap e375: 8 total, 8 up, 8 in 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:45.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:45.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-51"}]: dispatch 2024-03-30T14:04:45.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:04:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:04:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:45 smithi138 ceph-mon[33397]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:04:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:45 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T14:04:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:45 smithi138 ceph-mon[33397]: osdmap e376: 8 total, 8 up, 8 in 2024-03-30T14:04:46.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:45 smithi102 ceph-mon[29392]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:04:46.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:45 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T14:04:46.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:45 smithi102 ceph-mon[29392]: osdmap e376: 8 total, 8 up, 8 in 2024-03-30T14:04:46.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:45 smithi102 ceph-mon[37717]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:04:46.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:45 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T14:04:46.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:45 smithi102 ceph-mon[37717]: osdmap e376: 8 total, 8 up, 8 in 2024-03-30T14:04:46.920 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:04:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:47.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:47.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:46 smithi138 ceph-mon[33397]: osdmap e377: 8 total, 8 up, 8 in 2024-03-30T14:04:47.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:47.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[37717]: osdmap e377: 8 total, 8 up, 8 in 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[29392]: osdmap e377: 8 total, 8 up, 8 in 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:47.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: pgmap v529: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:04:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: osdmap e378: 8 total, 8 up, 8 in 2024-03-30T14:04:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:48.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-53"}]: dispatch 2024-03-30T14:04:48.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-53"}]: dispatch 2024-03-30T14:04:48.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: pgmap v529: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: osdmap e378: 8 total, 8 up, 8 in 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-53"}]: dispatch 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-53"}]: dispatch 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: pgmap v529: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:04:48.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:48.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:48.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: osdmap e378: 8 total, 8 up, 8 in 2024-03-30T14:04:48.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:48.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:48.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-53"}]: dispatch 2024-03-30T14:04:48.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-53"}]: dispatch 2024-03-30T14:04:49.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:48 smithi102 ceph-mon[29392]: osdmap e379: 8 total, 8 up, 8 in 2024-03-30T14:04:49.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:48 smithi102 ceph-mon[37717]: osdmap e379: 8 total, 8 up, 8 in 2024-03-30T14:04:49.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:48 smithi138 ceph-mon[33397]: osdmap e379: 8 total, 8 up, 8 in 2024-03-30T14:04:50.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:49 smithi138 ceph-mon[33397]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:04:50.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:49 smithi138 ceph-mon[33397]: osdmap e380: 8 total, 8 up, 8 in 2024-03-30T14:04:50.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:50.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[37717]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[37717]: osdmap e380: 8 total, 8 up, 8 in 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[29392]: pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[29392]: osdmap e380: 8 total, 8 up, 8 in 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:50 smithi138 ceph-mon[33397]: osdmap e381: 8 total, 8 up, 8 in 2024-03-30T14:04:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:04:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-55"}]: dispatch 2024-03-30T14:04:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-55"}]: dispatch 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[37717]: osdmap e381: 8 total, 8 up, 8 in 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-55"}]: dispatch 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-55"}]: dispatch 2024-03-30T14:04:51.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:51.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[29392]: osdmap e381: 8 total, 8 up, 8 in 2024-03-30T14:04:51.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:04:51.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:51.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:51.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-55"}]: dispatch 2024-03-30T14:04:51.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-55"}]: dispatch 2024-03-30T14:04:52.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:51 smithi138 ceph-mon[33397]: pgmap v535: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-30T14:04:52.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:51 smithi138 ceph-mon[33397]: osdmap e382: 8 total, 8 up, 8 in 2024-03-30T14:04:52.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:51 smithi102 ceph-mon[37717]: pgmap v535: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-30T14:04:52.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:51 smithi102 ceph-mon[37717]: osdmap e382: 8 total, 8 up, 8 in 2024-03-30T14:04:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:51 smithi102 ceph-mon[29392]: pgmap v535: 260 pgs: 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-30T14:04:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:51 smithi102 ceph-mon[29392]: osdmap e382: 8 total, 8 up, 8 in 2024-03-30T14:04:53.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:52 smithi138 ceph-mon[33397]: osdmap e383: 8 total, 8 up, 8 in 2024-03-30T14:04:53.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:53.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:52 smithi102 ceph-mon[29392]: osdmap e383: 8 total, 8 up, 8 in 2024-03-30T14:04:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:53.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:53.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:52 smithi102 ceph-mon[37717]: osdmap e383: 8 total, 8 up, 8 in 2024-03-30T14:04:53.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:53.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: pgmap v538: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:04:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: osdmap e384: 8 total, 8 up, 8 in 2024-03-30T14:04:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-57"}]: dispatch 2024-03-30T14:04:54.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-57"}]: dispatch 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: pgmap v538: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: osdmap e384: 8 total, 8 up, 8 in 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-57"}]: dispatch 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-57"}]: dispatch 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: pgmap v538: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:54.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: osdmap e384: 8 total, 8 up, 8 in 2024-03-30T14:04:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-57"}]: dispatch 2024-03-30T14:04:54.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-57"}]: dispatch 2024-03-30T14:04:55.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:54 smithi138 ceph-mon[33397]: osdmap e385: 8 total, 8 up, 8 in 2024-03-30T14:04:55.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:54 smithi102 ceph-mon[29392]: osdmap e385: 8 total, 8 up, 8 in 2024-03-30T14:04:55.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:54 smithi102 ceph-mon[37717]: osdmap e385: 8 total, 8 up, 8 in 2024-03-30T14:04:55.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:04:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:04:56.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[29392]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:56.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[29392]: osdmap e386: 8 total, 8 up, 8 in 2024-03-30T14:04:56.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:56.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:56.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[37717]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:56.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[37717]: osdmap e386: 8 total, 8 up, 8 in 2024-03-30T14:04:56.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:56.338 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:55 smithi138 ceph-mon[33397]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:04:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:55 smithi138 ceph-mon[33397]: osdmap e386: 8 total, 8 up, 8 in 2024-03-30T14:04:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:56.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:56.686 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:04:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:04:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:04:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: osdmap e387: 8 total, 8 up, 8 in 2024-03-30T14:04:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:04:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-59"}]: dispatch 2024-03-30T14:04:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-59"}]: dispatch 2024-03-30T14:04:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:57.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:56 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: osdmap e387: 8 total, 8 up, 8 in 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-59"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-59"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:04:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: osdmap e387: 8 total, 8 up, 8 in 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-59"}]: dispatch 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-59"}]: dispatch 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:04:57.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:04:58.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:57 smithi138 ceph-mon[33397]: pgmap v544: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:04:58.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:57 smithi138 ceph-mon[33397]: osdmap e388: 8 total, 8 up, 8 in 2024-03-30T14:04:58.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: pgmap v544: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:04:58.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[29392]: osdmap e388: 8 total, 8 up, 8 in 2024-03-30T14:04:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: pgmap v544: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:04:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:57 smithi102 ceph-mon[37717]: osdmap e388: 8 total, 8 up, 8 in 2024-03-30T14:04:59.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:59 smithi138 ceph-mon[33397]: osdmap e389: 8 total, 8 up, 8 in 2024-03-30T14:04:59.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:59.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:04:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:59.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:59 smithi102 ceph-mon[29392]: osdmap e389: 8 total, 8 up, 8 in 2024-03-30T14:04:59.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:59.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:04:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:59.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:59 smithi102 ceph-mon[37717]: osdmap e389: 8 total, 8 up, 8 in 2024-03-30T14:04:59.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:04:59.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:04:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:00.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:00 smithi138 ceph-mon[33397]: pgmap v547: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:05:00.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:00 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:00.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:00.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:00 smithi138 ceph-mon[33397]: osdmap e390: 8 total, 8 up, 8 in 2024-03-30T14:05:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:00.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[29392]: pgmap v547: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[29392]: osdmap e390: 8 total, 8 up, 8 in 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[37717]: pgmap v547: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T14:05:00.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[37717]: osdmap e390: 8 total, 8 up, 8 in 2024-03-30T14:05:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:01.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:01.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:01 smithi138 ceph-mon[33397]: osdmap e391: 8 total, 8 up, 8 in 2024-03-30T14:05:01.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:01.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:01.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-61"}]: dispatch 2024-03-30T14:05:01.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-61"}]: dispatch 2024-03-30T14:05:01.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[29392]: osdmap e391: 8 total, 8 up, 8 in 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-61"}]: dispatch 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-61"}]: dispatch 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[37717]: osdmap e391: 8 total, 8 up, 8 in 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-61"}]: dispatch 2024-03-30T14:05:01.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-61"}]: dispatch 2024-03-30T14:05:02.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:02 smithi138 ceph-mon[33397]: pgmap v550: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-30T14:05:02.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:02 smithi138 ceph-mon[33397]: osdmap e392: 8 total, 8 up, 8 in 2024-03-30T14:05:02.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:02 smithi102 ceph-mon[29392]: pgmap v550: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-30T14:05:02.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:02 smithi102 ceph-mon[29392]: osdmap e392: 8 total, 8 up, 8 in 2024-03-30T14:05:02.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:02 smithi102 ceph-mon[37717]: pgmap v550: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-30T14:05:02.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:02 smithi102 ceph-mon[37717]: osdmap e392: 8 total, 8 up, 8 in 2024-03-30T14:05:03.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:03 smithi138 ceph-mon[33397]: osdmap e393: 8 total, 8 up, 8 in 2024-03-30T14:05:03.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:03.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:03.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:03 smithi102 ceph-mon[29392]: osdmap e393: 8 total, 8 up, 8 in 2024-03-30T14:05:03.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:03.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:03.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:03 smithi102 ceph-mon[37717]: osdmap e393: 8 total, 8 up, 8 in 2024-03-30T14:05:03.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:03.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:04.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:04 smithi138 ceph-mon[33397]: pgmap v553: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:05:04.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:04.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:04 smithi138 ceph-mon[33397]: osdmap e394: 8 total, 8 up, 8 in 2024-03-30T14:05:04.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:04.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:04.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[29392]: pgmap v553: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[29392]: osdmap e394: 8 total, 8 up, 8 in 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[37717]: pgmap v553: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[37717]: osdmap e394: 8 total, 8 up, 8 in 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:04.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:05.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:05.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:05 smithi138 ceph-mon[33397]: osdmap e395: 8 total, 8 up, 8 in 2024-03-30T14:05:05.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:05.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:05 smithi102 ceph-mon[29392]: osdmap e395: 8 total, 8 up, 8 in 2024-03-30T14:05:05.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:05.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:05 smithi102 ceph-mon[37717]: osdmap e395: 8 total, 8 up, 8 in 2024-03-30T14:05:05.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:05:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:05:06.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:06 smithi138 ceph-mon[33397]: pgmap v556: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:06.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:06 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:06.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:06 smithi138 ceph-mon[33397]: osdmap e396: 8 total, 8 up, 8 in 2024-03-30T14:05:06.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:06 smithi102 ceph-mon[29392]: pgmap v556: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:06.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:06 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:06.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:06 smithi102 ceph-mon[29392]: osdmap e396: 8 total, 8 up, 8 in 2024-03-30T14:05:06.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:06 smithi102 ceph-mon[37717]: pgmap v556: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:06.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:06 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:06.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:06 smithi102 ceph-mon[37717]: osdmap e396: 8 total, 8 up, 8 in 2024-03-30T14:05:06.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:05:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:05:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:07.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:07.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:07 smithi138 ceph-mon[33397]: osdmap e397: 8 total, 8 up, 8 in 2024-03-30T14:05:07.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:07.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:07 smithi102 ceph-mon[29392]: osdmap e397: 8 total, 8 up, 8 in 2024-03-30T14:05:07.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:07.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:07 smithi102 ceph-mon[37717]: osdmap e397: 8 total, 8 up, 8 in 2024-03-30T14:05:08.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:08 smithi138 ceph-mon[33397]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:05:08.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:08 smithi138 ceph-mon[33397]: osdmap e398: 8 total, 8 up, 8 in 2024-03-30T14:05:08.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:08 smithi102 ceph-mon[29392]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:05:08.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:08 smithi102 ceph-mon[29392]: osdmap e398: 8 total, 8 up, 8 in 2024-03-30T14:05:08.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:08 smithi102 ceph-mon[37717]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:05:08.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:08 smithi102 ceph-mon[37717]: osdmap e398: 8 total, 8 up, 8 in 2024-03-30T14:05:09.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:09 smithi138 ceph-mon[33397]: osdmap e399: 8 total, 8 up, 8 in 2024-03-30T14:05:09.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:09 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:05:09.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:09 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:05:09.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:09 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:05:09.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[29392]: osdmap e399: 8 total, 8 up, 8 in 2024-03-30T14:05:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:05:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:05:09.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:05:09.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[37717]: osdmap e399: 8 total, 8 up, 8 in 2024-03-30T14:05:09.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:05:09.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:05:09.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:09 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:05:10.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:10 smithi138 ceph-mon[33397]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:05:10.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:05:10.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:10 smithi102 ceph-mon[29392]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:05:10.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:05:10.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:10 smithi102 ceph-mon[37717]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T14:05:10.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:05:12.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:12 smithi138 ceph-mon[33397]: pgmap v563: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-30T14:05:12.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:12 smithi138 ceph-mon[33397]: osdmap e400: 8 total, 8 up, 8 in 2024-03-30T14:05:12.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:12.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:12 smithi102 ceph-mon[29392]: pgmap v563: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-30T14:05:12.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:12 smithi102 ceph-mon[29392]: osdmap e400: 8 total, 8 up, 8 in 2024-03-30T14:05:12.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:12.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:12 smithi102 ceph-mon[37717]: pgmap v563: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-30T14:05:12.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:12 smithi102 ceph-mon[37717]: osdmap e400: 8 total, 8 up, 8 in 2024-03-30T14:05:12.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:14.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:14 smithi138 ceph-mon[33397]: pgmap v565: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-30T14:05:14.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:14 smithi102 ceph-mon[29392]: pgmap v565: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-30T14:05:14.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:14 smithi102 ceph-mon[37717]: pgmap v565: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-30T14:05:15.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:05:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:05:16.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:16 smithi138 ceph-mon[33397]: pgmap v566: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 561 B/s wr, 2 op/s 2024-03-30T14:05:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:16 smithi102 ceph-mon[29392]: pgmap v566: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 561 B/s wr, 2 op/s 2024-03-30T14:05:16.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:16 smithi102 ceph-mon[37717]: pgmap v566: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 561 B/s wr, 2 op/s 2024-03-30T14:05:16.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:05:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:05:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:17.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:17.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:17.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:18.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:18 smithi138 ceph-mon[33397]: pgmap v567: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 493 B/s wr, 2 op/s 2024-03-30T14:05:18.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:18 smithi102 ceph-mon[29392]: pgmap v567: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 493 B/s wr, 2 op/s 2024-03-30T14:05:18.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:18 smithi102 ceph-mon[37717]: pgmap v567: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 493 B/s wr, 2 op/s 2024-03-30T14:05:19.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:19 smithi138 ceph-mon[33397]: osdmap e401: 8 total, 8 up, 8 in 2024-03-30T14:05:19.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:19 smithi102 ceph-mon[29392]: osdmap e401: 8 total, 8 up, 8 in 2024-03-30T14:05:19.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:19 smithi102 ceph-mon[37717]: osdmap e401: 8 total, 8 up, 8 in 2024-03-30T14:05:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:20 smithi138 ceph-mon[33397]: pgmap v569: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:05:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:20 smithi102 ceph-mon[29392]: pgmap v569: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:05:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:20 smithi102 ceph-mon[37717]: pgmap v569: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:05:22.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:22 smithi138 ceph-mon[33397]: pgmap v570: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:22.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:22 smithi138 ceph-mon[33397]: osdmap e402: 8 total, 8 up, 8 in 2024-03-30T14:05:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:22 smithi102 ceph-mon[29392]: pgmap v570: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:22 smithi102 ceph-mon[29392]: osdmap e402: 8 total, 8 up, 8 in 2024-03-30T14:05:22.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:22 smithi102 ceph-mon[37717]: pgmap v570: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:22.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:22 smithi102 ceph-mon[37717]: osdmap e402: 8 total, 8 up, 8 in 2024-03-30T14:05:24.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:24 smithi138 ceph-mon[33397]: pgmap v572: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:24 smithi102 ceph-mon[29392]: pgmap v572: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:24.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:24 smithi102 ceph-mon[37717]: pgmap v572: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:25 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]: dispatch 2024-03-30T14:05:25.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:25 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]: dispatch 2024-03-30T14:05:25.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:25 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]: dispatch 2024-03-30T14:05:25.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:25 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]: dispatch 2024-03-30T14:05:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]: dispatch 2024-03-30T14:05:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]: dispatch 2024-03-30T14:05:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]: dispatch 2024-03-30T14:05:25.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]: dispatch 2024-03-30T14:05:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]: dispatch 2024-03-30T14:05:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]: dispatch 2024-03-30T14:05:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]: dispatch 2024-03-30T14:05:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:25 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]: dispatch 2024-03-30T14:05:25.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:05:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:05:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:26 smithi138 ceph-mon[33397]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]': finished 2024-03-30T14:05:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]': finished 2024-03-30T14:05:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:26 smithi138 ceph-mon[33397]: osdmap e403: 8 total, 8 up, 8 in 2024-03-30T14:05:26.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[29392]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]': finished 2024-03-30T14:05:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]': finished 2024-03-30T14:05:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[29392]: osdmap e403: 8 total, 8 up, 8 in 2024-03-30T14:05:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[37717]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:05:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.f", "id": [5, 0]}]': finished 2024-03-30T14:05:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "284.17", "id": [6, 4]}]': finished 2024-03-30T14:05:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-mon[37717]: osdmap e403: 8 total, 8 up, 8 in 2024-03-30T14:05:26.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:05:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:05:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:27 smithi138 ceph-mon[33397]: osdmap e404: 8 total, 8 up, 8 in 2024-03-30T14:05:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:27 smithi102 ceph-mon[29392]: osdmap e404: 8 total, 8 up, 8 in 2024-03-30T14:05:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:27 smithi102 ceph-mon[37717]: osdmap e404: 8 total, 8 up, 8 in 2024-03-30T14:05:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:28.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:28 smithi138 ceph-mon[33397]: pgmap v576: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:28 smithi102 ceph-mon[37717]: pgmap v576: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:28.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:28 smithi102 ceph-mon[29392]: pgmap v576: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:29.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:29 smithi138 ceph-mon[33397]: osdmap e405: 8 total, 8 up, 8 in 2024-03-30T14:05:29.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:29 smithi102 ceph-mon[37717]: osdmap e405: 8 total, 8 up, 8 in 2024-03-30T14:05:29.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:29 smithi102 ceph-mon[29392]: osdmap e405: 8 total, 8 up, 8 in 2024-03-30T14:05:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:30 smithi138 ceph-mon[33397]: pgmap v578: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:30.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:30 smithi102 ceph-mon[37717]: pgmap v578: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:30.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:30 smithi102 ceph-mon[29392]: pgmap v578: 260 pgs: 2 peering, 258 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:05:32.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:32 smithi138 ceph-mon[33397]: pgmap v579: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T14:05:32.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:32 smithi138 ceph-mon[33397]: osdmap e406: 8 total, 8 up, 8 in 2024-03-30T14:05:32.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:32 smithi102 ceph-mon[37717]: pgmap v579: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T14:05:32.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:32 smithi102 ceph-mon[37717]: osdmap e406: 8 total, 8 up, 8 in 2024-03-30T14:05:32.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:32 smithi102 ceph-mon[29392]: pgmap v579: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T14:05:32.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:32 smithi102 ceph-mon[29392]: osdmap e406: 8 total, 8 up, 8 in 2024-03-30T14:05:34.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:34 smithi138 ceph-mon[33397]: pgmap v581: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-30T14:05:34.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:34 smithi102 ceph-mon[29392]: pgmap v581: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-30T14:05:34.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:34 smithi102 ceph-mon[37717]: pgmap v581: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-30T14:05:35.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:05:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:05:36.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:36 smithi138 ceph-mon[33397]: pgmap v582: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:05:36.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:36 smithi102 ceph-mon[29392]: pgmap v582: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:05:36.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:36 smithi102 ceph-mon[37717]: pgmap v582: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:05:36.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:05:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:05:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:37.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:37.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:38.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:38 smithi102 ceph-mon[29392]: pgmap v583: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:05:38.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:38 smithi102 ceph-mon[37717]: pgmap v583: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:05:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:38 smithi138 ceph-mon[33397]: pgmap v583: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:05:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-63"}]: dispatch 2024-03-30T14:05:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-63"}]: dispatch 2024-03-30T14:05:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:39.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-63"}]: dispatch 2024-03-30T14:05:39.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-63"}]: dispatch 2024-03-30T14:05:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-63"}]: dispatch 2024-03-30T14:05:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-63"}]: dispatch 2024-03-30T14:05:40.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:40 smithi138 ceph-mon[33397]: pgmap v584: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:05:40.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:40 smithi138 ceph-mon[33397]: osdmap e407: 8 total, 8 up, 8 in 2024-03-30T14:05:40.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:40 smithi102 ceph-mon[29392]: pgmap v584: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:05:40.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:40 smithi102 ceph-mon[29392]: osdmap e407: 8 total, 8 up, 8 in 2024-03-30T14:05:40.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:40 smithi102 ceph-mon[37717]: pgmap v584: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:05:40.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:40 smithi102 ceph-mon[37717]: osdmap e407: 8 total, 8 up, 8 in 2024-03-30T14:05:41.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:41 smithi138 ceph-mon[33397]: osdmap e408: 8 total, 8 up, 8 in 2024-03-30T14:05:41.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:41.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:41.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:41 smithi102 ceph-mon[29392]: osdmap e408: 8 total, 8 up, 8 in 2024-03-30T14:05:41.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:41.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:41.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:41 smithi102 ceph-mon[37717]: osdmap e408: 8 total, 8 up, 8 in 2024-03-30T14:05:41.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:41.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: pgmap v587: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: osdmap e409: 8 total, 8 up, 8 in 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: pgmap v587: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: osdmap e409: 8 total, 8 up, 8 in 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: pgmap v587: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: osdmap e409: 8 total, 8 up, 8 in 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:42.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:05:42.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:43.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:43.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:43 smithi138 ceph-mon[33397]: osdmap e410: 8 total, 8 up, 8 in 2024-03-30T14:05:43.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:43.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:43 smithi102 ceph-mon[29392]: osdmap e410: 8 total, 8 up, 8 in 2024-03-30T14:05:43.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:05:43.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:43 smithi102 ceph-mon[37717]: osdmap e410: 8 total, 8 up, 8 in 2024-03-30T14:05:44.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:44 smithi138 ceph-mon[33397]: pgmap v590: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:44.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:44 smithi138 ceph-mon[33397]: osdmap e411: 8 total, 8 up, 8 in 2024-03-30T14:05:44.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:44 smithi102 ceph-mon[29392]: pgmap v590: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:44.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:44 smithi102 ceph-mon[29392]: osdmap e411: 8 total, 8 up, 8 in 2024-03-30T14:05:44.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:44 smithi102 ceph-mon[37717]: pgmap v590: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:44.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:44 smithi102 ceph-mon[37717]: osdmap e411: 8 total, 8 up, 8 in 2024-03-30T14:05:45.660 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:45 smithi138 ceph-mon[33397]: osdmap e412: 8 total, 8 up, 8 in 2024-03-30T14:05:45.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:45 smithi102 ceph-mon[29392]: osdmap e412: 8 total, 8 up, 8 in 2024-03-30T14:05:45.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:45 smithi102 ceph-mon[37717]: osdmap e412: 8 total, 8 up, 8 in 2024-03-30T14:05:45.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:05:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:05:46.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:46 smithi102 ceph-mon[29392]: pgmap v593: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:46.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:46 smithi102 ceph-mon[29392]: osdmap e413: 8 total, 8 up, 8 in 2024-03-30T14:05:46.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:46.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:46 smithi102 ceph-mon[37717]: pgmap v593: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:46.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:46 smithi102 ceph-mon[37717]: osdmap e413: 8 total, 8 up, 8 in 2024-03-30T14:05:46.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:46.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:46 smithi138 ceph-mon[33397]: pgmap v593: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:46.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:46 smithi138 ceph-mon[33397]: osdmap e413: 8 total, 8 up, 8 in 2024-03-30T14:05:46.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:46.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:05:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:05:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:47.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:47 smithi138 ceph-mon[33397]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-30T14:05:47.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:47 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:47.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:47 smithi102 ceph-mon[29392]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-30T14:05:47.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:47 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:47.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:47 smithi102 ceph-mon[37717]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-30T14:05:47.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:47 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:05:49.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:49 smithi102 ceph-mon[37717]: pgmap v596: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:05:49.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:49 smithi102 ceph-mon[29392]: pgmap v596: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:05:49.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:49 smithi138 ceph-mon[33397]: pgmap v596: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:05:51.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:51 smithi102 ceph-mon[29392]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-30T14:05:51.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:51 smithi102 ceph-mon[37717]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-30T14:05:51.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:51 smithi138 ceph-mon[33397]: pgmap v597: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-30T14:05:52.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:52 smithi138 ceph-mon[33397]: osdmap e414: 8 total, 8 up, 8 in 2024-03-30T14:05:52.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:52 smithi102 ceph-mon[29392]: osdmap e414: 8 total, 8 up, 8 in 2024-03-30T14:05:52.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:52 smithi102 ceph-mon[37717]: osdmap e414: 8 total, 8 up, 8 in 2024-03-30T14:05:53.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:53 smithi138 ceph-mon[33397]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-30T14:05:53.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:53 smithi102 ceph-mon[37717]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-30T14:05:53.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:53 smithi102 ceph-mon[29392]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-30T14:05:55.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:05:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:05:55.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:55 smithi138 ceph-mon[33397]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:05:55.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:55.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:55.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-65"}]: dispatch 2024-03-30T14:05:55.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-65"}]: dispatch 2024-03-30T14:05:55.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[29392]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:05:55.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:55.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:55.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-65"}]: dispatch 2024-03-30T14:05:55.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-65"}]: dispatch 2024-03-30T14:05:55.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[37717]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:05:55.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:55.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:55.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-65"}]: dispatch 2024-03-30T14:05:55.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-65"}]: dispatch 2024-03-30T14:05:56.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:56 smithi102 ceph-mon[29392]: osdmap e415: 8 total, 8 up, 8 in 2024-03-30T14:05:56.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:56.837 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:05:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:05:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:05:56.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:56 smithi102 ceph-mon[37717]: osdmap e415: 8 total, 8 up, 8 in 2024-03-30T14:05:56.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:56.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:56 smithi138 ceph-mon[33397]: osdmap e415: 8 total, 8 up, 8 in 2024-03-30T14:05:56.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:05:57.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:57 smithi138 ceph-mon[33397]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:57.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:57 smithi138 ceph-mon[33397]: osdmap e416: 8 total, 8 up, 8 in 2024-03-30T14:05:57.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:57.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:57.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[37717]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[37717]: osdmap e416: 8 total, 8 up, 8 in 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[29392]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[29392]: osdmap e416: 8 total, 8 up, 8 in 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:05:57.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:05:58.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:58.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:58 smithi138 ceph-mon[33397]: osdmap e417: 8 total, 8 up, 8 in 2024-03-30T14:05:58.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:58.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:58.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:58 smithi102 ceph-mon[29392]: osdmap e417: 8 total, 8 up, 8 in 2024-03-30T14:05:58.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:58.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:05:58.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:58 smithi102 ceph-mon[37717]: osdmap e417: 8 total, 8 up, 8 in 2024-03-30T14:05:58.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:05:59.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:59 smithi138 ceph-mon[33397]: pgmap v605: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:59.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:59 smithi138 ceph-mon[33397]: osdmap e418: 8 total, 8 up, 8 in 2024-03-30T14:05:59.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:59.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:59.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-67"}]: dispatch 2024-03-30T14:05:59.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:05:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-67"}]: dispatch 2024-03-30T14:05:59.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[37717]: pgmap v605: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:59.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[37717]: osdmap e418: 8 total, 8 up, 8 in 2024-03-30T14:05:59.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:59.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-67"}]: dispatch 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-67"}]: dispatch 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[29392]: pgmap v605: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[29392]: osdmap e418: 8 total, 8 up, 8 in 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-67"}]: dispatch 2024-03-30T14:05:59.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:05:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-67"}]: dispatch 2024-03-30T14:06:00.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:00 smithi138 ceph-mon[33397]: osdmap e419: 8 total, 8 up, 8 in 2024-03-30T14:06:00.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:00 smithi102 ceph-mon[29392]: osdmap e419: 8 total, 8 up, 8 in 2024-03-30T14:06:00.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:00 smithi102 ceph-mon[37717]: osdmap e419: 8 total, 8 up, 8 in 2024-03-30T14:06:01.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:01 smithi138 ceph-mon[33397]: pgmap v608: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:06:01.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:01 smithi138 ceph-mon[33397]: osdmap e420: 8 total, 8 up, 8 in 2024-03-30T14:06:01.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:01.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:01.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[37717]: pgmap v608: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:06:01.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[37717]: osdmap e420: 8 total, 8 up, 8 in 2024-03-30T14:06:01.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:01.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:01.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[29392]: pgmap v608: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:06:01.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[29392]: osdmap e420: 8 total, 8 up, 8 in 2024-03-30T14:06:01.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:01.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:02.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:02.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:02 smithi138 ceph-mon[33397]: osdmap e421: 8 total, 8 up, 8 in 2024-03-30T14:06:02.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:02.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:02.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:02.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-69"}]: dispatch 2024-03-30T14:06:02.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-69"}]: dispatch 2024-03-30T14:06:02.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:02.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[37717]: osdmap e421: 8 total, 8 up, 8 in 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-69"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-69"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[29392]: osdmap e421: 8 total, 8 up, 8 in 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-69"}]: dispatch 2024-03-30T14:06:02.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-69"}]: dispatch 2024-03-30T14:06:03.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:03 smithi138 ceph-mon[33397]: pgmap v611: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:06:03.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:03 smithi138 ceph-mon[33397]: osdmap e422: 8 total, 8 up, 8 in 2024-03-30T14:06:03.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:03 smithi102 ceph-mon[37717]: pgmap v611: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:06:03.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:03 smithi102 ceph-mon[37717]: osdmap e422: 8 total, 8 up, 8 in 2024-03-30T14:06:03.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:03 smithi102 ceph-mon[29392]: pgmap v611: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:06:03.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:03 smithi102 ceph-mon[29392]: osdmap e422: 8 total, 8 up, 8 in 2024-03-30T14:06:05.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:04 smithi138 ceph-mon[33397]: osdmap e423: 8 total, 8 up, 8 in 2024-03-30T14:06:05.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:05.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:04 smithi102 ceph-mon[29392]: osdmap e423: 8 total, 8 up, 8 in 2024-03-30T14:06:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:04 smithi102 ceph-mon[37717]: osdmap e423: 8 total, 8 up, 8 in 2024-03-30T14:06:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:05.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:06:05.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-mon[33397]: pgmap v614: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:05.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:05.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:05.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-mon[33397]: osdmap e424: 8 total, 8 up, 8 in 2024-03-30T14:06:05.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:06:05.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:06:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[29392]: pgmap v614: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[29392]: osdmap e424: 8 total, 8 up, 8 in 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[37717]: pgmap v614: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:06.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[37717]: osdmap e424: 8 total, 8 up, 8 in 2024-03-30T14:06:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:06:06.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:06:06.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:06.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:06:06.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:06 smithi102 ceph-mon[29392]: osdmap e425: 8 total, 8 up, 8 in 2024-03-30T14:06:06.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:06:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:06:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:06.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:06.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:06:06.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:06 smithi102 ceph-mon[37717]: osdmap e425: 8 total, 8 up, 8 in 2024-03-30T14:06:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:07.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:06:07.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:06 smithi138 ceph-mon[33397]: osdmap e425: 8 total, 8 up, 8 in 2024-03-30T14:06:08.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:07 smithi138 ceph-mon[33397]: pgmap v617: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:06:08.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:07 smithi138 ceph-mon[33397]: osdmap e426: 8 total, 8 up, 8 in 2024-03-30T14:06:08.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:07 smithi102 ceph-mon[29392]: pgmap v617: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:06:08.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:07 smithi102 ceph-mon[29392]: osdmap e426: 8 total, 8 up, 8 in 2024-03-30T14:06:08.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:07 smithi102 ceph-mon[37717]: pgmap v617: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:06:08.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:07 smithi102 ceph-mon[37717]: osdmap e426: 8 total, 8 up, 8 in 2024-03-30T14:06:10.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:09 smithi138 ceph-mon[33397]: pgmap v619: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 653 B/s wr, 1 op/s 2024-03-30T14:06:10.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:09 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:06:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:09 smithi102 ceph-mon[29392]: pgmap v619: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 653 B/s wr, 1 op/s 2024-03-30T14:06:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:09 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:06:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:09 smithi102 ceph-mon[37717]: pgmap v619: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 653 B/s wr, 1 op/s 2024-03-30T14:06:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:09 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:06:11.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:10 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:06:11.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:10 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:06:11.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:10 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:06:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:10 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:06:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:10 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:06:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:10 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:06:11.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:10 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:06:11.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:10 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:06:11.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:10 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:06:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:11 smithi138 ceph-mon[33397]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-30T14:06:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:11 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:11 smithi102 ceph-mon[29392]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-30T14:06:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:11 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:11 smithi102 ceph-mon[37717]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-30T14:06:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:11 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:13.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:14.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:13 smithi138 ceph-mon[33397]: pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-30T14:06:14.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:13 smithi102 ceph-mon[29392]: pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-30T14:06:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:13 smithi102 ceph-mon[37717]: pgmap v621: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-03-30T14:06:16.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:15 smithi138 ceph-mon[33397]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:06:16.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:06:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:06:16.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:15 smithi102 ceph-mon[29392]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:06:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:15 smithi102 ceph-mon[37717]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 706 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:06:16.797 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:06:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:06:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:17.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:17.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:18.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:17 smithi138 ceph-mon[33397]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-30T14:06:18.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:17 smithi102 ceph-mon[29392]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-30T14:06:18.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:17 smithi102 ceph-mon[37717]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-30T14:06:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:19 smithi138 ceph-mon[33397]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-30T14:06:20.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:19 smithi102 ceph-mon[29392]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-30T14:06:20.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:19 smithi102 ceph-mon[37717]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-30T14:06:22.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:21 smithi138 ceph-mon[33397]: pgmap v625: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:06:22.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:21 smithi102 ceph-mon[29392]: pgmap v625: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:06:22.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:21 smithi102 ceph-mon[37717]: pgmap v625: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-30T14:06:24.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:23 smithi138 ceph-mon[33397]: pgmap v626: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:24.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:23 smithi102 ceph-mon[37717]: pgmap v626: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:24.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:23 smithi102 ceph-mon[29392]: pgmap v626: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:26.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:06:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-mon[33397]: pgmap v627: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]: dispatch 2024-03-30T14:06:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]: dispatch 2024-03-30T14:06:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:25 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[37717]: pgmap v627: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:26.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]: dispatch 2024-03-30T14:06:26.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[29392]: pgmap v627: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:25 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]: dispatch 2024-03-30T14:06:26.821 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:06:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:27.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:27.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]': finished 2024-03-30T14:06:27.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]': finished 2024-03-30T14:06:27.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]': finished 2024-03-30T14:06:27.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:26 smithi138 ceph-mon[33397]: osdmap e427: 8 total, 8 up, 8 in 2024-03-30T14:06:27.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:27.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]': finished 2024-03-30T14:06:27.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]': finished 2024-03-30T14:06:27.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]': finished 2024-03-30T14:06:27.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[29392]: osdmap e427: 8 total, 8 up, 8 in 2024-03-30T14:06:27.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:27.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.6", "id": [6, 4]}]': finished 2024-03-30T14:06:27.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1a", "id": [6, 0]}]': finished 2024-03-30T14:06:27.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "288.1f", "id": [6, 0]}]': finished 2024-03-30T14:06:27.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:26 smithi102 ceph-mon[37717]: osdmap e427: 8 total, 8 up, 8 in 2024-03-30T14:06:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:27 smithi138 ceph-mon[33397]: pgmap v629: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-30T14:06:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:27 smithi138 ceph-mon[33397]: osdmap e428: 8 total, 8 up, 8 in 2024-03-30T14:06:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-71"}]: dispatch 2024-03-30T14:06:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-71"}]: dispatch 2024-03-30T14:06:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:28.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[29392]: pgmap v629: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-30T14:06:28.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[29392]: osdmap e428: 8 total, 8 up, 8 in 2024-03-30T14:06:28.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:28.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-71"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-71"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[37717]: pgmap v629: 260 pgs: 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[37717]: osdmap e428: 8 total, 8 up, 8 in 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-71"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-71"}]: dispatch 2024-03-30T14:06:28.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:28 smithi138 ceph-mon[33397]: osdmap e429: 8 total, 8 up, 8 in 2024-03-30T14:06:29.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:28 smithi102 ceph-mon[29392]: osdmap e429: 8 total, 8 up, 8 in 2024-03-30T14:06:29.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:28 smithi102 ceph-mon[37717]: osdmap e429: 8 total, 8 up, 8 in 2024-03-30T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:29 smithi138 ceph-mon[33397]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:29 smithi138 ceph-mon[33397]: osdmap e430: 8 total, 8 up, 8 in 2024-03-30T14:06:30.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:30.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:30.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[29392]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:30.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[29392]: osdmap e430: 8 total, 8 up, 8 in 2024-03-30T14:06:30.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:30.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:30.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[37717]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:30.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[37717]: osdmap e430: 8 total, 8 up, 8 in 2024-03-30T14:06:30.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:30.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:31.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:31.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:30 smithi138 ceph-mon[33397]: osdmap e431: 8 total, 8 up, 8 in 2024-03-30T14:06:31.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:31.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:31.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:30 smithi102 ceph-mon[29392]: osdmap e431: 8 total, 8 up, 8 in 2024-03-30T14:06:31.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:31.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:31.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:30 smithi102 ceph-mon[37717]: osdmap e431: 8 total, 8 up, 8 in 2024-03-30T14:06:31.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:32.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:31 smithi138 ceph-mon[33397]: pgmap v635: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:32.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:31 smithi138 ceph-mon[33397]: osdmap e432: 8 total, 8 up, 8 in 2024-03-30T14:06:32.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:31 smithi102 ceph-mon[37717]: pgmap v635: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:32.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:31 smithi102 ceph-mon[37717]: osdmap e432: 8 total, 8 up, 8 in 2024-03-30T14:06:32.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:31 smithi102 ceph-mon[29392]: pgmap v635: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:32.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:31 smithi102 ceph-mon[29392]: osdmap e432: 8 total, 8 up, 8 in 2024-03-30T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:32 smithi138 ceph-mon[33397]: osdmap e433: 8 total, 8 up, 8 in 2024-03-30T14:06:33.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:32 smithi102 ceph-mon[29392]: osdmap e433: 8 total, 8 up, 8 in 2024-03-30T14:06:33.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:32 smithi102 ceph-mon[37717]: osdmap e433: 8 total, 8 up, 8 in 2024-03-30T14:06:34.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:33 smithi138 ceph-mon[33397]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:06:34.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:33 smithi102 ceph-mon[29392]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:06:34.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:33 smithi102 ceph-mon[37717]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:35 smithi138 ceph-mon[33397]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 922 B/s wr, 1 op/s 2024-03-30T14:06:36.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:06:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:06:36.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:35 smithi102 ceph-mon[29392]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 922 B/s wr, 1 op/s 2024-03-30T14:06:36.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:35 smithi102 ceph-mon[37717]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 922 B/s wr, 1 op/s 2024-03-30T14:06:36.884 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:06:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:06:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:37.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:37.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:37.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:37 smithi138 ceph-mon[33397]: pgmap v640: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:06:38.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:37 smithi102 ceph-mon[37717]: pgmap v640: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:06:38.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:37 smithi102 ceph-mon[29392]: pgmap v640: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:06:40.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:39 smithi138 ceph-mon[33397]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:39 smithi102 ceph-mon[37717]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:40.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:39 smithi102 ceph-mon[29392]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:06:42.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:41 smithi138 ceph-mon[33397]: pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 214 B/s wr, 2 op/s 2024-03-30T14:06:42.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:41 smithi102 ceph-mon[37717]: pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 214 B/s wr, 2 op/s 2024-03-30T14:06:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:41 smithi102 ceph-mon[29392]: pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 214 B/s wr, 2 op/s 2024-03-30T14:06:43.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:43.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:43.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-73"}]: dispatch 2024-03-30T14:06:43.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-73"}]: dispatch 2024-03-30T14:06:43.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-73"}]: dispatch 2024-03-30T14:06:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-73"}]: dispatch 2024-03-30T14:06:43.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-73"}]: dispatch 2024-03-30T14:06:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-73"}]: dispatch 2024-03-30T14:06:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:44.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:43 smithi138 ceph-mon[33397]: pgmap v643: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 194 B/s wr, 2 op/s 2024-03-30T14:06:44.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:43 smithi138 ceph-mon[33397]: osdmap e434: 8 total, 8 up, 8 in 2024-03-30T14:06:44.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:43 smithi102 ceph-mon[29392]: pgmap v643: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 194 B/s wr, 2 op/s 2024-03-30T14:06:44.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:43 smithi102 ceph-mon[29392]: osdmap e434: 8 total, 8 up, 8 in 2024-03-30T14:06:44.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:43 smithi102 ceph-mon[37717]: pgmap v643: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 194 B/s wr, 2 op/s 2024-03-30T14:06:44.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:43 smithi102 ceph-mon[37717]: osdmap e434: 8 total, 8 up, 8 in 2024-03-30T14:06:45.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:44 smithi102 ceph-mon[29392]: osdmap e435: 8 total, 8 up, 8 in 2024-03-30T14:06:45.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:45.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:44 smithi102 ceph-mon[37717]: osdmap e435: 8 total, 8 up, 8 in 2024-03-30T14:06:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:45.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:44 smithi138 ceph-mon[33397]: osdmap e435: 8 total, 8 up, 8 in 2024-03-30T14:06:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:45.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:46.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:06:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:06:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:45 smithi138 ceph-mon[33397]: pgmap v646: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:45 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:45 smithi138 ceph-mon[33397]: osdmap e436: 8 total, 8 up, 8 in 2024-03-30T14:06:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:46.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[29392]: pgmap v646: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[29392]: osdmap e436: 8 total, 8 up, 8 in 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[37717]: pgmap v646: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[37717]: osdmap e436: 8 total, 8 up, 8 in 2024-03-30T14:06:46.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:46.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:06:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:06:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:46 smithi138 ceph-mon[33397]: osdmap e437: 8 total, 8 up, 8 in 2024-03-30T14:06:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:46 smithi102 ceph-mon[29392]: osdmap e437: 8 total, 8 up, 8 in 2024-03-30T14:06:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:47.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:46 smithi102 ceph-mon[37717]: osdmap e437: 8 total, 8 up, 8 in 2024-03-30T14:06:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:47 smithi138 ceph-mon[33397]: pgmap v649: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:48.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:47 smithi138 ceph-mon[33397]: osdmap e438: 8 total, 8 up, 8 in 2024-03-30T14:06:48.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:48.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:48.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-75"}]: dispatch 2024-03-30T14:06:48.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-75"}]: dispatch 2024-03-30T14:06:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[37717]: pgmap v649: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[37717]: osdmap e438: 8 total, 8 up, 8 in 2024-03-30T14:06:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-75"}]: dispatch 2024-03-30T14:06:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-75"}]: dispatch 2024-03-30T14:06:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[29392]: pgmap v649: 260 pgs: 260 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[29392]: osdmap e438: 8 total, 8 up, 8 in 2024-03-30T14:06:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-75"}]: dispatch 2024-03-30T14:06:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-75"}]: dispatch 2024-03-30T14:06:49.333 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:48 smithi138 ceph-mon[33397]: osdmap e439: 8 total, 8 up, 8 in 2024-03-30T14:06:49.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:48 smithi102 ceph-mon[29392]: osdmap e439: 8 total, 8 up, 8 in 2024-03-30T14:06:49.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:48 smithi102 ceph-mon[37717]: osdmap e439: 8 total, 8 up, 8 in 2024-03-30T14:06:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:49 smithi138 ceph-mon[33397]: pgmap v652: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:06:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:49 smithi138 ceph-mon[33397]: osdmap e440: 8 total, 8 up, 8 in 2024-03-30T14:06:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:50.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:50.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[29392]: pgmap v652: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:06:50.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[29392]: osdmap e440: 8 total, 8 up, 8 in 2024-03-30T14:06:50.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:50.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[37717]: pgmap v652: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:06:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[37717]: osdmap e440: 8 total, 8 up, 8 in 2024-03-30T14:06:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:50.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:51.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:51.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:50 smithi138 ceph-mon[33397]: osdmap e441: 8 total, 8 up, 8 in 2024-03-30T14:06:51.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:51.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:51.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:50 smithi102 ceph-mon[37717]: osdmap e441: 8 total, 8 up, 8 in 2024-03-30T14:06:51.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:51.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:51.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:50 smithi102 ceph-mon[29392]: osdmap e441: 8 total, 8 up, 8 in 2024-03-30T14:06:51.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:51 smithi138 ceph-mon[33397]: pgmap v655: 260 pgs: 13 creating+activating, 247 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:06:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:51 smithi138 ceph-mon[33397]: osdmap e442: 8 total, 8 up, 8 in 2024-03-30T14:06:52.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:52.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:52.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-77"}]: dispatch 2024-03-30T14:06:52.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-77"}]: dispatch 2024-03-30T14:06:52.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:51 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[37717]: pgmap v655: 260 pgs: 13 creating+activating, 247 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:06:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[37717]: osdmap e442: 8 total, 8 up, 8 in 2024-03-30T14:06:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-77"}]: dispatch 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-77"}]: dispatch 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[29392]: pgmap v655: 260 pgs: 13 creating+activating, 247 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[29392]: osdmap e442: 8 total, 8 up, 8 in 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-77"}]: dispatch 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-77"}]: dispatch 2024-03-30T14:06:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:51 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:06:53.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:53 smithi138 ceph-mon[33397]: osdmap e443: 8 total, 8 up, 8 in 2024-03-30T14:06:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:53 smithi102 ceph-mon[29392]: osdmap e443: 8 total, 8 up, 8 in 2024-03-30T14:06:53.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:53 smithi102 ceph-mon[37717]: osdmap e443: 8 total, 8 up, 8 in 2024-03-30T14:06:54.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:54 smithi138 ceph-mon[33397]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:06:54.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:54 smithi138 ceph-mon[33397]: osdmap e444: 8 total, 8 up, 8 in 2024-03-30T14:06:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:54.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:54.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[29392]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:06:54.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[29392]: osdmap e444: 8 total, 8 up, 8 in 2024-03-30T14:06:54.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:54.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:54.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[37717]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:06:54.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[37717]: osdmap e444: 8 total, 8 up, 8 in 2024-03-30T14:06:54.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:54.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:06:55.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:55.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:55 smithi138 ceph-mon[33397]: osdmap e445: 8 total, 8 up, 8 in 2024-03-30T14:06:55.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:55.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:55.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:55 smithi102 ceph-mon[29392]: osdmap e445: 8 total, 8 up, 8 in 2024-03-30T14:06:55.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:55.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:06:55.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:55 smithi102 ceph-mon[37717]: osdmap e445: 8 total, 8 up, 8 in 2024-03-30T14:06:55.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:06:56.029 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:06:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:06:56.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:56 smithi102 ceph-mon[29392]: pgmap v661: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:56.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:56 smithi102 ceph-mon[29392]: osdmap e446: 8 total, 8 up, 8 in 2024-03-30T14:06:56.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:56 smithi102 ceph-mon[37717]: pgmap v661: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:56.337 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:56 smithi102 ceph-mon[37717]: osdmap e446: 8 total, 8 up, 8 in 2024-03-30T14:06:56.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:56 smithi138 ceph-mon[33397]: pgmap v661: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:06:56.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:56 smithi138 ceph-mon[33397]: osdmap e446: 8 total, 8 up, 8 in 2024-03-30T14:06:56.685 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:06:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:06:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:06:57.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:57.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:57 smithi138 ceph-mon[33397]: osdmap e447: 8 total, 8 up, 8 in 2024-03-30T14:06:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.8"}]: dispatch 2024-03-30T14:06:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:57 smithi138 ceph-mon[33397]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.8"}]: dispatch 2024-03-30T14:06:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:57 smithi138 ceph-mon[33397]: 292.8 deep-scrub starts 2024-03-30T14:06:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:57 smithi138 ceph-mon[33397]: 292.8 deep-scrub ok 2024-03-30T14:06:57.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:57.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:57.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[37717]: osdmap e447: 8 total, 8 up, 8 in 2024-03-30T14:06:57.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.8"}]: dispatch 2024-03-30T14:06:57.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[37717]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.8"}]: dispatch 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[37717]: 292.8 deep-scrub starts 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[37717]: 292.8 deep-scrub ok 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[29392]: osdmap e447: 8 total, 8 up, 8 in 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.8"}]: dispatch 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[29392]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "292.8"}]: dispatch 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[29392]: 292.8 deep-scrub starts 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[29392]: 292.8 deep-scrub ok 2024-03-30T14:06:57.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:06:58.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:06:58 smithi138 ceph-mon[33397]: pgmap v664: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-30T14:06:58.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:06:58 smithi102 ceph-mon[29392]: pgmap v664: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-30T14:06:58.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:06:58 smithi102 ceph-mon[37717]: pgmap v664: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-30T14:07:00.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:00 smithi138 ceph-mon[33397]: pgmap v665: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 947 B/s wr, 134 op/s 2024-03-30T14:07:00.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:00 smithi102 ceph-mon[29392]: pgmap v665: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 947 B/s wr, 134 op/s 2024-03-30T14:07:00.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:00 smithi102 ceph-mon[37717]: pgmap v665: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 947 B/s wr, 134 op/s 2024-03-30T14:07:02.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:02 smithi138 ceph-mon[33397]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 960 B/s wr, 115 op/s 2024-03-30T14:07:02.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:02 smithi102 ceph-mon[29392]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 960 B/s wr, 115 op/s 2024-03-30T14:07:02.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:02 smithi102 ceph-mon[37717]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 960 B/s wr, 115 op/s 2024-03-30T14:07:04.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:04 smithi138 ceph-mon[33397]: pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 767 B/s wr, 92 op/s 2024-03-30T14:07:04.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:04 smithi102 ceph-mon[29392]: pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 767 B/s wr, 92 op/s 2024-03-30T14:07:04.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:04 smithi102 ceph-mon[37717]: pgmap v667: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 767 B/s wr, 92 op/s 2024-03-30T14:07:06.049 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:07:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:07:06.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:06 smithi138 ceph-mon[33397]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 654 B/s wr, 78 op/s 2024-03-30T14:07:06.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:06 smithi102 ceph-mon[29392]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 654 B/s wr, 78 op/s 2024-03-30T14:07:06.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:06 smithi102 ceph-mon[37717]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 654 B/s wr, 78 op/s 2024-03-30T14:07:06.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:07:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:07:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:07:07.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:07.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:07.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:08.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:08 smithi138 ceph-mon[33397]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 986 B/s rd, 98 B/s wr, 1 op/s 2024-03-30T14:07:08.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:08 smithi102 ceph-mon[29392]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 986 B/s rd, 98 B/s wr, 1 op/s 2024-03-30T14:07:08.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:08 smithi102 ceph-mon[37717]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 986 B/s rd, 98 B/s wr, 1 op/s 2024-03-30T14:07:10.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:10 smithi138 ceph-mon[33397]: pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-30T14:07:10.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:10 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:07:10.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:10 smithi102 ceph-mon[37717]: pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-30T14:07:10.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:10 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:07:10.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:10 smithi102 ceph-mon[29392]: pgmap v670: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-30T14:07:10.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:10 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:07:11.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:07:11.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:07:11.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:07:11.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:07:11.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:07:11.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:07:12.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:12 smithi138 ceph-mon[33397]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-30T14:07:12.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:12.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:12 smithi102 ceph-mon[29392]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-30T14:07:12.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:12.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:12 smithi102 ceph-mon[37717]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-30T14:07:12.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:14.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:14 smithi138 ceph-mon[33397]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:07:14.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:14 smithi102 ceph-mon[29392]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:07:14.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:14 smithi102 ceph-mon[37717]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:07:16.062 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:07:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:07:16.431 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:07:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:07:16.068+0000 7f78817ca700 -1 snap_mapper.add_oid found existing snaps mapped on 292:178275fc:test-rados-api-smithi102-75117-80::foo:2, removing 2024-03-30T14:07:16.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:16 smithi138 ceph-mon[33397]: pgmap v673: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:07:16.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:16.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:16.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-79"}]: dispatch 2024-03-30T14:07:16.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-79"}]: dispatch 2024-03-30T14:07:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[29392]: pgmap v673: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:07:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-79"}]: dispatch 2024-03-30T14:07:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-79"}]: dispatch 2024-03-30T14:07:16.436 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T14:07:16.068+0000 7f6b59c8b700 -1 snap_mapper.add_oid found existing snaps mapped on 292:178275fc:test-rados-api-smithi102-75117-80::foo:2, removing 2024-03-30T14:07:16.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[37717]: pgmap v673: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:07:16.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:16.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:16.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-79"}]: dispatch 2024-03-30T14:07:16.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-79"}]: dispatch 2024-03-30T14:07:16.436 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1[47444]: 2024-03-30T14:07:16.068+0000 7f0e3ed5d700 -1 snap_mapper.add_oid found existing snaps mapped on 292:178275fc:test-rados-api-smithi102-75117-80::foo:2, removing 2024-03-30T14:07:16.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:07:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:07:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:07:17.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:17.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:17.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:18 smithi138 ceph-mon[33397]: pgmap v674: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:18.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:18 smithi138 ceph-mon[33397]: osdmap e448: 8 total, 8 up, 8 in 2024-03-30T14:07:18.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:18 smithi102 ceph-mon[29392]: pgmap v674: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:18.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:18 smithi102 ceph-mon[29392]: osdmap e448: 8 total, 8 up, 8 in 2024-03-30T14:07:18.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:18 smithi102 ceph-mon[37717]: pgmap v674: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:18.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:18 smithi102 ceph-mon[37717]: osdmap e448: 8 total, 8 up, 8 in 2024-03-30T14:07:19.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:19 smithi138 ceph-mon[33397]: osdmap e449: 8 total, 8 up, 8 in 2024-03-30T14:07:19.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:19.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:19 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:19.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:19 smithi102 ceph-mon[29392]: osdmap e449: 8 total, 8 up, 8 in 2024-03-30T14:07:19.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:19.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:19 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:19.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:19 smithi102 ceph-mon[37717]: osdmap e449: 8 total, 8 up, 8 in 2024-03-30T14:07:19.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:19.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:19 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:20.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:20 smithi138 ceph-mon[33397]: pgmap v677: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:07:20.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:20 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:20 smithi138 ceph-mon[33397]: osdmap e450: 8 total, 8 up, 8 in 2024-03-30T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[29392]: pgmap v677: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:07:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[29392]: osdmap e450: 8 total, 8 up, 8 in 2024-03-30T14:07:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[37717]: pgmap v677: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:07:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[37717]: osdmap e450: 8 total, 8 up, 8 in 2024-03-30T14:07:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:21.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:21.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:21 smithi138 ceph-mon[33397]: osdmap e451: 8 total, 8 up, 8 in 2024-03-30T14:07:21.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:21.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:21.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:21.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[29392]: osdmap e451: 8 total, 8 up, 8 in 2024-03-30T14:07:21.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:21.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:21.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:21.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[37717]: osdmap e451: 8 total, 8 up, 8 in 2024-03-30T14:07:21.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:21.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:22.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:22 smithi138 ceph-mon[33397]: pgmap v680: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-30T14:07:22.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:22.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:22.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:22 smithi138 ceph-mon[33397]: osdmap e452: 8 total, 8 up, 8 in 2024-03-30T14:07:22.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:22 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[29392]: pgmap v680: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-30T14:07:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[29392]: osdmap e452: 8 total, 8 up, 8 in 2024-03-30T14:07:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:22.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[37717]: pgmap v680: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-30T14:07:22.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:22.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:22.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[37717]: osdmap e452: 8 total, 8 up, 8 in 2024-03-30T14:07:22.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:22 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:23.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:23.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:23 smithi138 ceph-mon[33397]: osdmap e453: 8 total, 8 up, 8 in 2024-03-30T14:07:23.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:23.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:23.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:23.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[29392]: osdmap e453: 8 total, 8 up, 8 in 2024-03-30T14:07:23.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:23.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:23.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:23.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[37717]: osdmap e453: 8 total, 8 up, 8 in 2024-03-30T14:07:23.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:23.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:24.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:24 smithi138 ceph-mon[33397]: pgmap v683: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:07:24.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:24.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:24 smithi138 ceph-mon[33397]: osdmap e454: 8 total, 8 up, 8 in 2024-03-30T14:07:24.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T14:07:24.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T14:07:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[29392]: pgmap v683: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:07:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[29392]: osdmap e454: 8 total, 8 up, 8 in 2024-03-30T14:07:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T14:07:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T14:07:24.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[37717]: pgmap v683: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:07:24.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:24.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[37717]: osdmap e454: 8 total, 8 up, 8 in 2024-03-30T14:07:24.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T14:07:24.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T14:07:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-30T14:07:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:25 smithi138 ceph-mon[33397]: osdmap e455: 8 total, 8 up, 8 in 2024-03-30T14:07:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T14:07:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T14:07:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-30T14:07:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[29392]: osdmap e455: 8 total, 8 up, 8 in 2024-03-30T14:07:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T14:07:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T14:07:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-30T14:07:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[37717]: osdmap e455: 8 total, 8 up, 8 in 2024-03-30T14:07:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T14:07:25.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T14:07:26.148 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:07:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:07:26.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:26 smithi138 ceph-mon[33397]: pgmap v686: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:26.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-30T14:07:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:26 smithi138 ceph-mon[33397]: osdmap e456: 8 total, 8 up, 8 in 2024-03-30T14:07:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:26.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[29392]: pgmap v686: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:26.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-30T14:07:26.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[29392]: osdmap e456: 8 total, 8 up, 8 in 2024-03-30T14:07:26.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:26.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:26.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[37717]: pgmap v686: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-30T14:07:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[37717]: osdmap e456: 8 total, 8 up, 8 in 2024-03-30T14:07:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:26.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:07:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:07:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: osdmap e457: 8 total, 8 up, 8 in 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-81"}]: dispatch 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-81"}]: dispatch 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: osdmap e457: 8 total, 8 up, 8 in 2024-03-30T14:07:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-81"}]: dispatch 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-81"}]: dispatch 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: osdmap e457: 8 total, 8 up, 8 in 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:27.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-81"}]: dispatch 2024-03-30T14:07:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-81"}]: dispatch 2024-03-30T14:07:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:27.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:28 smithi138 ceph-mon[33397]: pgmap v689: 260 pgs: 260 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-30T14:07:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:28 smithi138 ceph-mon[33397]: osdmap e458: 8 total, 8 up, 8 in 2024-03-30T14:07:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:28 smithi102 ceph-mon[29392]: pgmap v689: 260 pgs: 260 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-30T14:07:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:28 smithi102 ceph-mon[29392]: osdmap e458: 8 total, 8 up, 8 in 2024-03-30T14:07:28.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:28 smithi102 ceph-mon[37717]: pgmap v689: 260 pgs: 260 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-30T14:07:28.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:28 smithi102 ceph-mon[37717]: osdmap e458: 8 total, 8 up, 8 in 2024-03-30T14:07:29.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:29 smithi138 ceph-mon[33397]: osdmap e459: 8 total, 8 up, 8 in 2024-03-30T14:07:29.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:29.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:29.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:29 smithi102 ceph-mon[29392]: osdmap e459: 8 total, 8 up, 8 in 2024-03-30T14:07:29.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:29.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:29.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:29 smithi102 ceph-mon[37717]: osdmap e459: 8 total, 8 up, 8 in 2024-03-30T14:07:29.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:29.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:30 smithi138 ceph-mon[33397]: pgmap v692: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-30T14:07:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:30 smithi138 ceph-mon[33397]: osdmap e460: 8 total, 8 up, 8 in 2024-03-30T14:07:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:30.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[29392]: pgmap v692: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-30T14:07:30.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:30.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[29392]: osdmap e460: 8 total, 8 up, 8 in 2024-03-30T14:07:30.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:30.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:30.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:30.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[37717]: pgmap v692: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-30T14:07:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[37717]: osdmap e460: 8 total, 8 up, 8 in 2024-03-30T14:07:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:31.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:31.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:31.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: osdmap e461: 8 total, 8 up, 8 in 2024-03-30T14:07:31.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: osdmap e462: 8 total, 8 up, 8 in 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: osdmap e461: 8 total, 8 up, 8 in 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: osdmap e462: 8 total, 8 up, 8 in 2024-03-30T14:07:31.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:31.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:31.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:31.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:31.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: osdmap e461: 8 total, 8 up, 8 in 2024-03-30T14:07:31.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:31.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:31.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: osdmap e462: 8 total, 8 up, 8 in 2024-03-30T14:07:31.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:31.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:32 smithi138 ceph-mon[33397]: pgmap v695: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-30T14:07:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:32 smithi138 ceph-mon[33397]: osdmap e463: 8 total, 8 up, 8 in 2024-03-30T14:07:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[37717]: pgmap v695: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-30T14:07:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[37717]: osdmap e463: 8 total, 8 up, 8 in 2024-03-30T14:07:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:32.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[29392]: pgmap v695: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-30T14:07:32.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:32.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[29392]: osdmap e463: 8 total, 8 up, 8 in 2024-03-30T14:07:32.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:32.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:34.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:34 smithi138 ceph-mon[33397]: pgmap v698: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T14:07:34.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:34.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:34 smithi138 ceph-mon[33397]: osdmap e464: 8 total, 8 up, 8 in 2024-03-30T14:07:34.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:34 smithi102 ceph-mon[37717]: pgmap v698: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T14:07:34.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:34.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:34 smithi102 ceph-mon[37717]: osdmap e464: 8 total, 8 up, 8 in 2024-03-30T14:07:34.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:34 smithi102 ceph-mon[29392]: pgmap v698: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T14:07:34.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:34.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:34 smithi102 ceph-mon[29392]: osdmap e464: 8 total, 8 up, 8 in 2024-03-30T14:07:35.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:35 smithi138 ceph-mon[33397]: osdmap e465: 8 total, 8 up, 8 in 2024-03-30T14:07:35.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:35 smithi102 ceph-mon[29392]: osdmap e465: 8 total, 8 up, 8 in 2024-03-30T14:07:35.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:35 smithi102 ceph-mon[37717]: osdmap e465: 8 total, 8 up, 8 in 2024-03-30T14:07:36.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:07:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:07:36.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[29392]: pgmap v701: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[29392]: osdmap e466: 8 total, 8 up, 8 in 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-83"}]: dispatch 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-83"}]: dispatch 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[37717]: pgmap v701: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[37717]: osdmap e466: 8 total, 8 up, 8 in 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:36.506 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:36.507 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-83"}]: dispatch 2024-03-30T14:07:36.507 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-83"}]: dispatch 2024-03-30T14:07:36.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:36 smithi138 ceph-mon[33397]: pgmap v701: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:36.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:36 smithi138 ceph-mon[33397]: osdmap e466: 8 total, 8 up, 8 in 2024-03-30T14:07:36.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:36.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:36.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-83"}]: dispatch 2024-03-30T14:07:36.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-83"}]: dispatch 2024-03-30T14:07:36.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:07:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:07:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:07:37.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:37.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:37 smithi138 ceph-mon[33397]: osdmap e467: 8 total, 8 up, 8 in 2024-03-30T14:07:37.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:37.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:37 smithi102 ceph-mon[29392]: osdmap e467: 8 total, 8 up, 8 in 2024-03-30T14:07:37.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:37.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:37 smithi102 ceph-mon[37717]: osdmap e467: 8 total, 8 up, 8 in 2024-03-30T14:07:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:38 smithi138 ceph-mon[33397]: pgmap v704: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T14:07:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:38 smithi138 ceph-mon[33397]: osdmap e468: 8 total, 8 up, 8 in 2024-03-30T14:07:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:38.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[29392]: pgmap v704: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T14:07:38.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[29392]: osdmap e468: 8 total, 8 up, 8 in 2024-03-30T14:07:38.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:38.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:38.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[37717]: pgmap v704: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T14:07:38.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[37717]: osdmap e468: 8 total, 8 up, 8 in 2024-03-30T14:07:38.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:38.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:39.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:39.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: osdmap e469: 8 total, 8 up, 8 in 2024-03-30T14:07:39.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: osdmap e470: 8 total, 8 up, 8 in 2024-03-30T14:07:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:39.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:39 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: osdmap e469: 8 total, 8 up, 8 in 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: osdmap e470: 8 total, 8 up, 8 in 2024-03-30T14:07:39.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: osdmap e469: 8 total, 8 up, 8 in 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: osdmap e470: 8 total, 8 up, 8 in 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:39.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:39 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:40.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:40 smithi138 ceph-mon[33397]: pgmap v707: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T14:07:40.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:40.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:40 smithi138 ceph-mon[33397]: osdmap e471: 8 total, 8 up, 8 in 2024-03-30T14:07:40.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:40.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:40.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[29392]: pgmap v707: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T14:07:40.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:40.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[29392]: osdmap e471: 8 total, 8 up, 8 in 2024-03-30T14:07:40.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:40.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:40.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[37717]: pgmap v707: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T14:07:40.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_tier","val": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:07:40.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[37717]: osdmap e471: 8 total, 8 up, 8 in 2024-03-30T14:07:40.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:40.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:42.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:42 smithi138 ceph-mon[33397]: pgmap v710: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:42.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:42.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:42 smithi138 ceph-mon[33397]: osdmap e472: 8 total, 8 up, 8 in 2024-03-30T14:07:42.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:42.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[29392]: pgmap v710: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[29392]: osdmap e472: 8 total, 8 up, 8 in 2024-03-30T14:07:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:42.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[37717]: pgmap v710: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[37717]: osdmap e472: 8 total, 8 up, 8 in 2024-03-30T14:07:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:42.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:43.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:43.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:43 smithi138 ceph-mon[33397]: osdmap e473: 8 total, 8 up, 8 in 2024-03-30T14:07:43.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:43.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:43 smithi102 ceph-mon[29392]: osdmap e473: 8 total, 8 up, 8 in 2024-03-30T14:07:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:43.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:43 smithi102 ceph-mon[37717]: osdmap e473: 8 total, 8 up, 8 in 2024-03-30T14:07:44.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:44 smithi138 ceph-mon[33397]: pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:44.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:44 smithi138 ceph-mon[33397]: osdmap e474: 8 total, 8 up, 8 in 2024-03-30T14:07:44.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:44 smithi102 ceph-mon[29392]: pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:44.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:44 smithi102 ceph-mon[29392]: osdmap e474: 8 total, 8 up, 8 in 2024-03-30T14:07:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:44 smithi102 ceph-mon[37717]: pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:44 smithi102 ceph-mon[37717]: osdmap e474: 8 total, 8 up, 8 in 2024-03-30T14:07:45.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:45 smithi138 ceph-mon[33397]: osdmap e475: 8 total, 8 up, 8 in 2024-03-30T14:07:45.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:45.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:45.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-85"}]: dispatch 2024-03-30T14:07:45.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-85"}]: dispatch 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[29392]: osdmap e475: 8 total, 8 up, 8 in 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-85"}]: dispatch 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-85"}]: dispatch 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[37717]: osdmap e475: 8 total, 8 up, 8 in 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:45.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:45.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-85"}]: dispatch 2024-03-30T14:07:45.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-85"}]: dispatch 2024-03-30T14:07:46.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:07:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:07:46.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:46 smithi102 ceph-mon[29392]: pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:46.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:46 smithi102 ceph-mon[29392]: osdmap e476: 8 total, 8 up, 8 in 2024-03-30T14:07:46.507 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:46.507 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:46 smithi102 ceph-mon[37717]: pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:46.507 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:46 smithi102 ceph-mon[37717]: osdmap e476: 8 total, 8 up, 8 in 2024-03-30T14:07:46.507 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:46.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:46 smithi138 ceph-mon[33397]: pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:46.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:46 smithi138 ceph-mon[33397]: osdmap e476: 8 total, 8 up, 8 in 2024-03-30T14:07:46.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:46.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:07:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:07:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:07:47.280 INFO:tasks.workunit.client.0.smithi102.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18337 ms) 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22713 ms) 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3023 ms) 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3022 ms) 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3019 ms) 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.281 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3027 ms) 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4025 ms) 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38172 ms) 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16430 ms) 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.282 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4031 ms) 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3021 ms) 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25148 ms) 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15081 ms) 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5060 ms) 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-30T14:07:47.283 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4028 ms) 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: waiting for scrubs... 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: done waiting 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25088 ms) 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10080 ms) 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9037 ms) 2024-03-30T14:07:47.284 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-30T14:07:47.285 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:07:47.285 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9055 ms) 2024-03-30T14:07:47.285 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-30T14:07:47.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:47 smithi138 ceph-mon[33397]: osdmap e477: 8 total, 8 up, 8 in 2024-03-30T14:07:47.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:47.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:47.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:47 smithi138 ceph-mon[33397]: pgmap v719: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T14:07:47.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[29392]: osdmap e477: 8 total, 8 up, 8 in 2024-03-30T14:07:47.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:47.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:47.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[29392]: pgmap v719: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T14:07:47.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[37717]: osdmap e477: 8 total, 8 up, 8 in 2024-03-30T14:07:47.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:47.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:47.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:47 smithi102 ceph-mon[37717]: pgmap v719: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T14:07:48.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:48 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:48.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:48.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:48 smithi138 ceph-mon[33397]: osdmap e478: 8 total, 8 up, 8 in 2024-03-30T14:07:48.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[37717]: osdmap e478: 8 total, 8 up, 8 in 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[29392]: osdmap e478: 8 total, 8 up, 8 in 2024-03-30T14:07:48.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:49.583 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:49 smithi138 ceph-mon[33397]: osdmap e479: 8 total, 8 up, 8 in 2024-03-30T14:07:49.583 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:49.583 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:49.583 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:49 smithi138 ceph-mon[33397]: pgmap v722: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T14:07:49.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[37717]: osdmap e479: 8 total, 8 up, 8 in 2024-03-30T14:07:49.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:49.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:49.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[37717]: pgmap v722: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T14:07:49.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[29392]: osdmap e479: 8 total, 8 up, 8 in 2024-03-30T14:07:49.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:49.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:49.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:49 smithi102 ceph-mon[29392]: pgmap v722: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T14:07:50.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:50.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:50 smithi138 ceph-mon[33397]: osdmap e480: 8 total, 8 up, 8 in 2024-03-30T14:07:50.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]: dispatch 2024-03-30T14:07:50.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]: dispatch 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[37717]: osdmap e480: 8 total, 8 up, 8 in 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]: dispatch 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]: dispatch 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[29392]: osdmap e480: 8 total, 8 up, 8 in 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]: dispatch 2024-03-30T14:07:50.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]: dispatch 2024-03-30T14:07:51.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]': finished 2024-03-30T14:07:51.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:51.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:51 smithi138 ceph-mon[33397]: osdmap e481: 8 total, 8 up, 8 in 2024-03-30T14:07:51.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:51.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:51 smithi138 ceph-mon[33397]: pgmap v725: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]': finished 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[29392]: osdmap e481: 8 total, 8 up, 8 in 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[29392]: pgmap v725: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_tier","val": "test-rados-api-smithi102-75117-89-test-flush"}]': finished 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:51.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[37717]: osdmap e481: 8 total, 8 up, 8 in 2024-03-30T14:07:51.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:07:51.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:51 smithi102 ceph-mon[37717]: pgmap v725: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:52.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:52.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:52 smithi138 ceph-mon[33397]: osdmap e482: 8 total, 8 up, 8 in 2024-03-30T14:07:52.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:52.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:52.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:52.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[29392]: osdmap e482: 8 total, 8 up, 8 in 2024-03-30T14:07:52.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:52.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:52.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:07:52.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[37717]: osdmap e482: 8 total, 8 up, 8 in 2024-03-30T14:07:52.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:52.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:07:53.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:53.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:53 smithi138 ceph-mon[33397]: osdmap e483: 8 total, 8 up, 8 in 2024-03-30T14:07:53.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:53 smithi138 ceph-mon[33397]: pgmap v728: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:53.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:53.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:53 smithi102 ceph-mon[29392]: osdmap e483: 8 total, 8 up, 8 in 2024-03-30T14:07:53.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:53 smithi102 ceph-mon[29392]: pgmap v728: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:53.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:07:53.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:53 smithi102 ceph-mon[37717]: osdmap e483: 8 total, 8 up, 8 in 2024-03-30T14:07:53.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:53 smithi102 ceph-mon[37717]: pgmap v728: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:07:54.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:54 smithi138 ceph-mon[33397]: osdmap e484: 8 total, 8 up, 8 in 2024-03-30T14:07:54.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:54.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:54.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-87"}]: dispatch 2024-03-30T14:07:54.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-87"}]: dispatch 2024-03-30T14:07:54.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[29392]: osdmap e484: 8 total, 8 up, 8 in 2024-03-30T14:07:54.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:54.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:54.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-87"}]: dispatch 2024-03-30T14:07:54.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-87"}]: dispatch 2024-03-30T14:07:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[37717]: osdmap e484: 8 total, 8 up, 8 in 2024-03-30T14:07:54.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:07:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-87"}]: dispatch 2024-03-30T14:07:54.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-87"}]: dispatch 2024-03-30T14:07:55.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:55 smithi138 ceph-mon[33397]: osdmap e485: 8 total, 8 up, 8 in 2024-03-30T14:07:55.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:55 smithi138 ceph-mon[33397]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:55.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:55 smithi102 ceph-mon[29392]: osdmap e485: 8 total, 8 up, 8 in 2024-03-30T14:07:55.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:55 smithi102 ceph-mon[29392]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:55.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:55 smithi102 ceph-mon[37717]: osdmap e485: 8 total, 8 up, 8 in 2024-03-30T14:07:55.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:55 smithi102 ceph-mon[37717]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:07:56.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:07:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:07:56.586 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:07:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[37717]: osdmap e486: 8 total, 8 up, 8 in 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[29392]: osdmap e486: 8 total, 8 up, 8 in 2024-03-30T14:07:56.587 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:56.588 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:56.588 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:56 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:07:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:56 smithi138 ceph-mon[33397]: osdmap e486: 8 total, 8 up, 8 in 2024-03-30T14:07:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:56 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:07:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:07:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:57 smithi138 ceph-mon[33397]: osdmap e487: 8 total, 8 up, 8 in 2024-03-30T14:07:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:57.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:57.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:57.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:57 smithi138 ceph-mon[33397]: pgmap v734: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:07:57.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[37717]: osdmap e487: 8 total, 8 up, 8 in 2024-03-30T14:07:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[37717]: pgmap v734: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:07:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:57.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:07:57.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[29392]: osdmap e487: 8 total, 8 up, 8 in 2024-03-30T14:07:57.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:07:57.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:57.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:07:57.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[29392]: pgmap v734: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:07:57.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:07:58.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:58.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:58 smithi138 ceph-mon[33397]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T14:07:58.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:58.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:58 smithi102 ceph-mon[29392]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T14:07:58.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:07:58.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:58 smithi102 ceph-mon[37717]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T14:07:59.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:59 smithi138 ceph-mon[33397]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T14:07:59.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:07:59 smithi138 ceph-mon[33397]: pgmap v737: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:07:59.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:59 smithi102 ceph-mon[37717]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T14:07:59.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:07:59 smithi102 ceph-mon[37717]: pgmap v737: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:07:59.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:59 smithi102 ceph-mon[29392]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T14:07:59.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:07:59 smithi102 ceph-mon[29392]: pgmap v737: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:08:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:00 smithi138 ceph-mon[33397]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T14:08:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-90"}]: dispatch 2024-03-30T14:08:00.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-90"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[37717]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-90"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-90"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[29392]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-90"}]: dispatch 2024-03-30T14:08:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-90"}]: dispatch 2024-03-30T14:08:01.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:01 smithi138 ceph-mon[33397]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T14:08:01.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:01 smithi138 ceph-mon[33397]: pgmap v740: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:08:01.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:01 smithi102 ceph-mon[29392]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T14:08:01.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:01 smithi102 ceph-mon[29392]: pgmap v740: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:08:01.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:01 smithi102 ceph-mon[37717]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T14:08:01.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:01 smithi102 ceph-mon[37717]: pgmap v740: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:08:02.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:02 smithi138 ceph-mon[33397]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T14:08:02.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:02.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:02 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:02.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:02 smithi102 ceph-mon[29392]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T14:08:02.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:02.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:02 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:02.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:02 smithi102 ceph-mon[37717]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T14:08:02.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:02.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:02 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:03.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:03.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:03 smithi138 ceph-mon[33397]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T14:08:03.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:03.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:03 smithi138 ceph-mon[33397]: pgmap v743: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:08:03.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:03.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[29392]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T14:08:03.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:03.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[29392]: pgmap v743: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:08:03.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:03.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[37717]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T14:08:03.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:03.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:03 smithi102 ceph-mon[37717]: pgmap v743: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:08:04.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:04 smithi138 ceph-mon[33397]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T14:08:04.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:04 smithi102 ceph-mon[29392]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T14:08:04.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:04 smithi102 ceph-mon[37717]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T14:08:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:05 smithi138 ceph-mon[33397]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T14:08:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:05 smithi138 ceph-mon[33397]: pgmap v746: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-92"}]: dispatch 2024-03-30T14:08:05.682 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-92"}]: dispatch 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[29392]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[29392]: pgmap v746: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-92"}]: dispatch 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-92"}]: dispatch 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[37717]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T14:08:05.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[37717]: pgmap v746: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:05.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:05.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:05.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-92"}]: dispatch 2024-03-30T14:08:05.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-92"}]: dispatch 2024-03-30T14:08:06.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:08:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:08:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:06 smithi138 ceph-mon[33397]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T14:08:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:06 smithi102 ceph-mon[29392]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T14:08:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:06.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:06 smithi102 ceph-mon[37717]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T14:08:06.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:06.685 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:08:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:08:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:08:07.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:07 smithi138 ceph-mon[33397]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 760 B/s wr, 2 op/s 2024-03-30T14:08:07.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:07 smithi138 ceph-mon[33397]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T14:08:07.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:07.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:07 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[37717]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 760 B/s wr, 2 op/s 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[37717]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[29392]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 760 B/s wr, 2 op/s 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[29392]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:07.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:07 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:08.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:08.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:08 smithi138 ceph-mon[33397]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T14:08:08.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:08.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:08.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:08 smithi102 ceph-mon[37717]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T14:08:08.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:08.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:08.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:08 smithi102 ceph-mon[29392]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T14:08:08.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:09.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:09 smithi138 ceph-mon[33397]: pgmap v751: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 764 B/s wr, 2 op/s 2024-03-30T14:08:09.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:09 smithi138 ceph-mon[33397]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T14:08:09.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:09 smithi102 ceph-mon[37717]: pgmap v751: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 764 B/s wr, 2 op/s 2024-03-30T14:08:09.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:09 smithi102 ceph-mon[37717]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T14:08:09.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:09 smithi102 ceph-mon[29392]: pgmap v751: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 764 B/s wr, 2 op/s 2024-03-30T14:08:09.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:09 smithi102 ceph-mon[29392]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T14:08:10.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:10 smithi138 ceph-mon[33397]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T14:08:10.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:10 smithi102 ceph-mon[29392]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T14:08:10.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:10 smithi102 ceph-mon[37717]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T14:08:11.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: pgmap v754: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-30T14:08:11.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:08:11.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi102 to 3248M 2024-03-30T14:08:11.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: Adjusting osd_memory_target on smithi138 to 3248M 2024-03-30T14:08:11.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:08:11.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:08:11.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: pgmap v754: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-30T14:08:11.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:08:11.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi102 to 3248M 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: Adjusting osd_memory_target on smithi138 to 3248M 2024-03-30T14:08:11.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: pgmap v754: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.937 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi102 to 3248M 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: Adjusting osd_memory_target on smithi138 to 3248M 2024-03-30T14:08:11.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:11.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:08:11.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:08:11.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:08:12.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:12.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:12.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:13.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:13 smithi138 ceph-mon[33397]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-30T14:08:13.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:13 smithi102 ceph-mon[29392]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-30T14:08:13.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:13 smithi102 ceph-mon[37717]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-30T14:08:15.813 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:15 smithi138 ceph-mon[33397]: pgmap v756: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:08:15.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:15 smithi102 ceph-mon[29392]: pgmap v756: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:08:15.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:15 smithi102 ceph-mon[37717]: pgmap v756: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T14:08:16.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:08:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:08:16.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:16.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:16.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:16.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:08:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:08:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:08:17.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:17 smithi138 ceph-mon[33397]: pgmap v757: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:08:17.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:17 smithi102 ceph-mon[29392]: pgmap v757: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:08:17.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:17 smithi102 ceph-mon[37717]: pgmap v757: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:08:19.733 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:19 smithi138 ceph-mon[33397]: pgmap v758: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 821 B/s wr, 3 op/s 2024-03-30T14:08:19.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:19 smithi102 ceph-mon[29392]: pgmap v758: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 821 B/s wr, 3 op/s 2024-03-30T14:08:19.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:19 smithi102 ceph-mon[37717]: pgmap v758: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 821 B/s wr, 3 op/s 2024-03-30T14:08:20.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:20 smithi138 ceph-mon[33397]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T14:08:20.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:20 smithi102 ceph-mon[29392]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T14:08:20.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:20 smithi102 ceph-mon[37717]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T14:08:21.431 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:08:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=cleanup t=2024-03-30T14:08:21.031658752Z level=info msg="Completed cleanup jobs" duration=2.456288ms 2024-03-30T14:08:21.431 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:08:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=sqlstore.transactions t=2024-03-30T14:08:21.040033468Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T14:08:21.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:21 smithi138 ceph-mon[33397]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:21.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:21 smithi102 ceph-mon[29392]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:21.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:21 smithi102 ceph-mon[37717]: pgmap v760: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:22.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:22 smithi138 ceph-mon[33397]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T14:08:22.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:22 smithi102 ceph-mon[29392]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T14:08:22.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:22 smithi102 ceph-mon[37717]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T14:08:23.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:23 smithi138 ceph-mon[33397]: pgmap v762: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:23.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:23 smithi102 ceph-mon[29392]: pgmap v762: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:23.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:23 smithi102 ceph-mon[37717]: pgmap v762: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:25.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:25 smithi138 ceph-mon[33397]: pgmap v763: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:25.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:08:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:08:25.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:25 smithi102 ceph-mon[29392]: pgmap v763: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:25.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:25 smithi102 ceph-mon[37717]: pgmap v763: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:26.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:08:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]: dispatch 2024-03-30T14:08:26.837 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:26 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]: dispatch 2024-03-30T14:08:26.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:26.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:26 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]: dispatch 2024-03-30T14:08:26.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:26 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]: dispatch 2024-03-30T14:08:26.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]: dispatch 2024-03-30T14:08:26.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:26 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]: dispatch 2024-03-30T14:08:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:27 smithi138 ceph-mon[33397]: pgmap v764: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]': finished 2024-03-30T14:08:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]': finished 2024-03-30T14:08:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:27 smithi138 ceph-mon[33397]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T14:08:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:27.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[29392]: pgmap v764: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:27.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]': finished 2024-03-30T14:08:27.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]': finished 2024-03-30T14:08:27.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[29392]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T14:08:27.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:27.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[37717]: pgmap v764: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:08:27.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.0", "id": [6, 3]}]': finished 2024-03-30T14:08:27.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "300.7", "id": [1, 2]}]': finished 2024-03-30T14:08:27.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[37717]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T14:08:27.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:28.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:28 smithi138 ceph-mon[33397]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T14:08:28.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:28 smithi102 ceph-mon[29392]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T14:08:28.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:28 smithi102 ceph-mon[37717]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T14:08:29.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:29 smithi138 ceph-mon[33397]: pgmap v767: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 753 B/s rd, 0 op/s 2024-03-30T14:08:29.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:29.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:29.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:29 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-94"}]: dispatch 2024-03-30T14:08:29.932 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-94"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[29392]: pgmap v767: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 753 B/s rd, 0 op/s 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-94"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-94"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[37717]: pgmap v767: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 753 B/s rd, 0 op/s 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-94"}]: dispatch 2024-03-30T14:08:29.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-94"}]: dispatch 2024-03-30T14:08:31.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:30 smithi138 ceph-mon[33397]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T14:08:31.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:30 smithi102 ceph-mon[29392]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T14:08:31.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:30 smithi102 ceph-mon[37717]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:31 smithi138 ceph-mon[33397]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:31 smithi138 ceph-mon[33397]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[37717]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[37717]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[29392]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[29392]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:32.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:32 smithi138 ceph-mon[33397]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:33.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:33.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:32 smithi102 ceph-mon[29392]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T14:08:33.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:33.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:33.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:32 smithi102 ceph-mon[37717]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T14:08:33.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:33 smithi138 ceph-mon[33397]: pgmap v772: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:33 smithi138 ceph-mon[33397]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T14:08:34.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:33 smithi102 ceph-mon[29392]: pgmap v772: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:34.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:33 smithi102 ceph-mon[29392]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T14:08:34.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:33 smithi102 ceph-mon[37717]: pgmap v772: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:34.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:33 smithi102 ceph-mon[37717]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T14:08:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:35 smithi138 ceph-mon[33397]: pgmap v774: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 215 B/s wr, 1 op/s 2024-03-30T14:08:36.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:08:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:08:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:35 smithi102 ceph-mon[37717]: pgmap v774: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 215 B/s wr, 1 op/s 2024-03-30T14:08:36.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:35 smithi102 ceph-mon[29392]: pgmap v774: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 215 B/s wr, 1 op/s 2024-03-30T14:08:36.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:36.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:08:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:08:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:08:37.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:37 smithi138 ceph-mon[33397]: pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-03-30T14:08:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:37 smithi102 ceph-mon[29392]: pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-03-30T14:08:38.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:37 smithi102 ceph-mon[37717]: pgmap v775: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-03-30T14:08:40.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:39 smithi138 ceph-mon[33397]: pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 661 B/s wr, 2 op/s 2024-03-30T14:08:40.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:39 smithi102 ceph-mon[29392]: pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 661 B/s wr, 2 op/s 2024-03-30T14:08:40.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:39 smithi102 ceph-mon[37717]: pgmap v776: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 661 B/s wr, 2 op/s 2024-03-30T14:08:42.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:41 smithi138 ceph-mon[33397]: pgmap v777: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 585 B/s wr, 2 op/s 2024-03-30T14:08:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:41 smithi102 ceph-mon[29392]: pgmap v777: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 585 B/s wr, 2 op/s 2024-03-30T14:08:42.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:41 smithi102 ceph-mon[37717]: pgmap v777: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 585 B/s wr, 2 op/s 2024-03-30T14:08:43.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:43.181 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:08:42 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:08:42.735+0000 7f3df69ef700 -1 snap_mapper.add_oid found existing snaps mapped on 112:3734c4fd:test-rados-api-smithi102-75117-97::foo:21, removing 2024-03-30T14:08:43.181 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:08:42 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:08:42.735+0000 7fdb3509b700 -1 snap_mapper.add_oid found existing snaps mapped on 112:3734c4fd:test-rados-api-smithi102-75117-97::foo:21, removing 2024-03-30T14:08:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:43.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:43.185 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:08:42 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1[47444]: 2024-03-30T14:08:42.735+0000 7f0e38d51700 -1 snap_mapper.add_oid found existing snaps mapped on 112:3734c4fd:test-rados-api-smithi102-75117-97::foo:21, removing 2024-03-30T14:08:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:44 smithi138 ceph-mon[33397]: pgmap v778: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:08:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-96"}]: dispatch 2024-03-30T14:08:44.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-96"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[37717]: pgmap v778: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-96"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-96"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[29392]: pgmap v778: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-96"}]: dispatch 2024-03-30T14:08:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-96"}]: dispatch 2024-03-30T14:08:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:45 smithi138 ceph-mon[33397]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T14:08:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:45 smithi138 ceph-mon[33397]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T14:08:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:45.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[37717]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T14:08:45.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[37717]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T14:08:45.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:45.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:45.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[29392]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T14:08:45.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[29392]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T14:08:45.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:45.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:46.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:08:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:08:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:46 smithi138 ceph-mon[33397]: pgmap v780: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-30T14:08:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:46 smithi138 ceph-mon[33397]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T14:08:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:46 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:46.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[37717]: pgmap v780: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-30T14:08:46.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:46.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[37717]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T14:08:46.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:46.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:46.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[29392]: pgmap v780: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-30T14:08:46.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[29392]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T14:08:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:46.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:08:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:08:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:08:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:08:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]: dispatch 2024-03-30T14:08:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:47 smithi138 ceph-mon[33397]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T14:08:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]: dispatch 2024-03-30T14:08:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:08:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]: dispatch 2024-03-30T14:08:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[37717]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T14:08:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]: dispatch 2024-03-30T14:08:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:08:47.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]: dispatch 2024-03-30T14:08:47.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[29392]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T14:08:47.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]: dispatch 2024-03-30T14:08:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:48 smithi138 ceph-mon[33397]: pgmap v783: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:08:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:48 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:08:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]': finished 2024-03-30T14:08:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:48 smithi138 ceph-mon[33397]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T14:08:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:48.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[37717]: pgmap v783: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]': finished 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[37717]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[29392]: pgmap v783: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-98", "mode": "writeback"}]': finished 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[29392]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:49 smithi138 ceph-mon[33397]: pgmap v786: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]': finished 2024-03-30T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:49 smithi138 ceph-mon[33397]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[37717]: pgmap v786: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]': finished 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[37717]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[29392]: pgmap v786: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-98"}]': finished 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[29392]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:08:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:08:51.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:50 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:08:51.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:08:51.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:50 smithi138 ceph-mon[33397]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T14:08:51.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T14:08:51.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[29392]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:08:51.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[37717]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T14:08:51.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T14:08:51.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:51 smithi138 ceph-mon[33397]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]': finished 2024-03-30T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:51 smithi138 ceph-mon[33397]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[29392]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]': finished 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[29392]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[37717]: pgmap v789: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_count","val": "1"}]': finished 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[37717]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T14:08:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:08:52.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:08:53.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:08:53.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:52 smithi138 ceph-mon[33397]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T14:08:53.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T14:08:53.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T14:08:53.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]': finished 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[29392]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]': finished 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[37717]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T14:08:53.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-98","var": "target_max_objects","val": "250"}]': finished 2024-03-30T14:08:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:53 smithi138 ceph-mon[33397]: pgmap v792: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:53 smithi138 ceph-mon[33397]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T14:08:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:53 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[37717]: pgmap v792: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[37717]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[29392]: pgmap v792: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[29392]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:54.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:53 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:08:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:08:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:54 smithi138 ceph-mon[33397]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T14:08:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:54 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[37717]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[29392]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T14:08:55.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:54 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]: dispatch 2024-03-30T14:08:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:55 smithi102 ceph-mon[29392]: pgmap v795: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:55 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]': finished 2024-03-30T14:08:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:55 smithi102 ceph-mon[29392]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T14:08:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:55 smithi102 ceph-mon[37717]: pgmap v795: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:56.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:55 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]': finished 2024-03-30T14:08:56.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:55 smithi102 ceph-mon[37717]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T14:08:56.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:08:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:08:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:55 smithi138 ceph-mon[33397]: pgmap v795: 260 pgs: 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:08:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:55 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-98"}]': finished 2024-03-30T14:08:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:55 smithi138 ceph-mon[33397]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T14:08:56.816 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:08:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:08:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:08:57.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:56 smithi138 ceph-mon[33397]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T14:08:57.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:57.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:56 smithi102 ceph-mon[29392]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T14:08:57.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:57.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:56 smithi102 ceph-mon[37717]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T14:08:57.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:08:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:57 smithi138 ceph-mon[33397]: pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:08:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:57 smithi138 ceph-mon[33397]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T14:08:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:57 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[29392]: pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[29392]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[37717]: pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[37717]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:08:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:08:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:58 smithi138 ceph-mon[33397]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T14:08:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:58 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[29392]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[37717]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:08:59.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:58 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:09:00.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:59 smithi138 ceph-mon[33397]: pgmap v801: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:00.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:09:00.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:00.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:59 smithi138 ceph-mon[33397]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T14:09:00.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:08:59 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[37717]: pgmap v801: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[37717]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[29392]: pgmap v801: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[29392]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T14:09:00.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:08:59 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]': finished 2024-03-30T14:09:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:00 smithi138 ceph-mon[33397]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T14:09:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]: dispatch 2024-03-30T14:09:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]: dispatch 2024-03-30T14:09:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:00 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:00 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]': finished 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]': finished 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[37717]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]: dispatch 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]: dispatch 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]': finished 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-6", "overlaypool": "test-rados-api-smithi102-75117-100"}]': finished 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[29392]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]: dispatch 2024-03-30T14:09:01.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]: dispatch 2024-03-30T14:09:01.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:01.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:00 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-100", "mode": "writeback"}]': finished 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: pgmap v804: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: osdmap e526: 8 total, 8 up, 8 in 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: osdmap e527: 8 total, 8 up, 8 in 2024-03-30T14:09:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:01 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: pgmap v804: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: osdmap e526: 8 total, 8 up, 8 in 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: osdmap e527: 8 total, 8 up, 8 in 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: pgmap v804: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: osdmap e526: 8 total, 8 up, 8 in 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:02.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:09:02.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:02.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: osdmap e527: 8 total, 8 up, 8 in 2024-03-30T14:09:02.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:01 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:03 smithi138 ceph-mon[33397]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:09:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:03 smithi138 ceph-mon[33397]: osdmap e528: 8 total, 8 up, 8 in 2024-03-30T14:09:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:03 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[37717]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[37717]: osdmap e528: 8 total, 8 up, 8 in 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[29392]: pgmap v807: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[29392]: osdmap e528: 8 total, 8 up, 8 in 2024-03-30T14:09:04.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:03 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:04 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:09:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:04 smithi138 ceph-mon[33397]: osdmap e529: 8 total, 8 up, 8 in 2024-03-30T14:09:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T14:09:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:04 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T14:09:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:09:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[29392]: osdmap e529: 8 total, 8 up, 8 in 2024-03-30T14:09:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T14:09:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T14:09:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:09:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[37717]: osdmap e529: 8 total, 8 up, 8 in 2024-03-30T14:09:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T14:09:05.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:04 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T14:09:06.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:09:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:09:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:05 smithi138 ceph-mon[33397]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-30T14:09:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:05 smithi138 ceph-mon[33397]: osdmap e530: 8 total, 8 up, 8 in 2024-03-30T14:09:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:06.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:05 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[37717]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[37717]: osdmap e530: 8 total, 8 up, 8 in 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[29392]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[29392]: osdmap e530: 8 total, 8 up, 8 in 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:05 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:06.875 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:09:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:09:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:09:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:06 smithi138 ceph-mon[33397]: osdmap e531: 8 total, 8 up, 8 in 2024-03-30T14:09:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:06 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]': finished 2024-03-30T14:09:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:06 smithi138 ceph-mon[33397]: osdmap e532: 8 total, 8 up, 8 in 2024-03-30T14:09:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:09:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[29392]: osdmap e531: 8 total, 8 up, 8 in 2024-03-30T14:09:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]': finished 2024-03-30T14:09:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[29392]: osdmap e532: 8 total, 8 up, 8 in 2024-03-30T14:09:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]': finished 2024-03-30T14:09:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[37717]: osdmap e531: 8 total, 8 up, 8 in 2024-03-30T14:09:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]: dispatch 2024-03-30T14:09:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-100"}]': finished 2024-03-30T14:09:07.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:06 smithi102 ceph-mon[37717]: osdmap e532: 8 total, 8 up, 8 in 2024-03-30T14:09:08.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:07 smithi138 ceph-mon[33397]: pgmap v813: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:09:08.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:07 smithi102 ceph-mon[29392]: pgmap v813: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:09:08.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:07 smithi102 ceph-mon[37717]: pgmap v813: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:09:09.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:08 smithi138 ceph-mon[33397]: osdmap e533: 8 total, 8 up, 8 in 2024-03-30T14:09:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:08 smithi102 ceph-mon[29392]: osdmap e533: 8 total, 8 up, 8 in 2024-03-30T14:09:09.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:08 smithi102 ceph-mon[37717]: osdmap e533: 8 total, 8 up, 8 in 2024-03-30T14:09:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:09 smithi138 ceph-mon[33397]: pgmap v816: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:09 smithi138 ceph-mon[33397]: osdmap e534: 8 total, 8 up, 8 in 2024-03-30T14:09:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[37717]: pgmap v816: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[37717]: osdmap e534: 8 total, 8 up, 8 in 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[29392]: pgmap v816: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[29392]: osdmap e534: 8 total, 8 up, 8 in 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:10 smithi138 ceph-mon[33397]: osdmap e535: 8 total, 8 up, 8 in 2024-03-30T14:09:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:09:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:09:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[29392]: osdmap e535: 8 total, 8 up, 8 in 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[37717]: osdmap e535: 8 total, 8 up, 8 in 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:09:11.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: pgmap v819: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: osdmap e536: 8 total, 8 up, 8 in 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:09:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:11 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: pgmap v819: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: osdmap e536: 8 total, 8 up, 8 in 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: pgmap v819: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:09:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: osdmap e536: 8 total, 8 up, 8 in 2024-03-30T14:09:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T14:09:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:09:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:09:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:09:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:11 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:09:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:12 smithi138 ceph-mon[33397]: osdmap e537: 8 total, 8 up, 8 in 2024-03-30T14:09:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:09:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:09:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:12 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:09:13.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:12 smithi138 ceph-mon[33397]: osdmap e538: 8 total, 8 up, 8 in 2024-03-30T14:09:13.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[29392]: osdmap e537: 8 total, 8 up, 8 in 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[29392]: osdmap e538: 8 total, 8 up, 8 in 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[37717]: osdmap e537: 8 total, 8 up, 8 in 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T14:09:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:12 smithi102 ceph-mon[37717]: osdmap e538: 8 total, 8 up, 8 in 2024-03-30T14:09:14.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:13 smithi138 ceph-mon[33397]: pgmap v822: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:14.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:14.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:14.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-102"}]: dispatch 2024-03-30T14:09:14.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:13 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-102"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[29392]: pgmap v822: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-102"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-102"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[37717]: pgmap v822: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-6"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1715047076' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-102"}]: dispatch 2024-03-30T14:09:14.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:13 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-6", "tierpool": "test-rados-api-smithi102-75117-102"}]: dispatch 2024-03-30T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:14 smithi138 ceph-mon[33397]: osdmap e539: 8 total, 8 up, 8 in 2024-03-30T14:09:15.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:14 smithi102 ceph-mon[29392]: osdmap e539: 8 total, 8 up, 8 in 2024-03-30T14:09:15.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:14 smithi102 ceph-mon[37717]: osdmap e539: 8 total, 8 up, 8 in 2024-03-30T14:09:16.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:09:16.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: pgmap v825: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:16.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: osdmap e540: 8 total, 8 up, 8 in 2024-03-30T14:09:16.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: pgmap v825: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: osdmap e540: 8 total, 8 up, 8 in 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: pgmap v825: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:16.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: osdmap e540: 8 total, 8 up, 8 in 2024-03-30T14:09:16.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:16.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:16.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:16.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:09:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:16 smithi138 ceph-mon[33397]: osdmap e541: 8 total, 8 up, 8 in 2024-03-30T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:17.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[29392]: osdmap e541: 8 total, 8 up, 8 in 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[37717]: osdmap e541: 8 total, 8 up, 8 in 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:17.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:17 smithi138 ceph-mon[33397]: pgmap v828: 196 pgs: 196 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:17 smithi138 ceph-mon[33397]: osdmap e542: 8 total, 8 up, 8 in 2024-03-30T14:09:18.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:17 smithi102 ceph-mon[29392]: pgmap v828: 196 pgs: 196 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:18.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:17 smithi102 ceph-mon[29392]: osdmap e542: 8 total, 8 up, 8 in 2024-03-30T14:09:18.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:17 smithi102 ceph-mon[37717]: pgmap v828: 196 pgs: 196 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:18.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:17 smithi102 ceph-mon[37717]: osdmap e542: 8 total, 8 up, 8 in 2024-03-30T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:18 smithi138 ceph-mon[33397]: osdmap e543: 8 total, 8 up, 8 in 2024-03-30T14:09:19.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:19.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:18 smithi102 ceph-mon[29392]: osdmap e543: 8 total, 8 up, 8 in 2024-03-30T14:09:19.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi102-75117-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:19.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:18 smithi102 ceph-mon[37717]: osdmap e543: 8 total, 8 up, 8 in 2024-03-30T14:09:20.332 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:19 smithi138 ceph-mon[33397]: pgmap v831: 204 pgs: 8 unknown, 196 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:20.332 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:19 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:20.332 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:19 smithi138 ceph-mon[33397]: osdmap e544: 8 total, 8 up, 8 in 2024-03-30T14:09:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:19 smithi102 ceph-mon[29392]: pgmap v831: 204 pgs: 8 unknown, 196 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:19 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:20.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:19 smithi102 ceph-mon[29392]: osdmap e544: 8 total, 8 up, 8 in 2024-03-30T14:09:20.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:19 smithi102 ceph-mon[37717]: pgmap v831: 204 pgs: 8 unknown, 196 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:20.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:19 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:20.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:19 smithi102 ceph-mon[37717]: osdmap e544: 8 total, 8 up, 8 in 2024-03-30T14:09:21.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:20 smithi138 ceph-mon[33397]: osdmap e545: 8 total, 8 up, 8 in 2024-03-30T14:09:21.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:21.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:20 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:21.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:20 smithi102 ceph-mon[37717]: osdmap e545: 8 total, 8 up, 8 in 2024-03-30T14:09:21.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:21.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:20 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:21.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:20 smithi102 ceph-mon[29392]: osdmap e545: 8 total, 8 up, 8 in 2024-03-30T14:09:21.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:21.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:20 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:22.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:21 smithi138 ceph-mon[33397]: pgmap v834: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:22.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:21 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:22.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:21 smithi138 ceph-mon[33397]: osdmap e546: 8 total, 8 up, 8 in 2024-03-30T14:09:22.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:21 smithi102 ceph-mon[29392]: pgmap v834: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:22.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:21 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:21 smithi102 ceph-mon[29392]: osdmap e546: 8 total, 8 up, 8 in 2024-03-30T14:09:22.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:21 smithi102 ceph-mon[37717]: pgmap v834: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:22.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:21 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:22.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:21 smithi102 ceph-mon[37717]: osdmap e546: 8 total, 8 up, 8 in 2024-03-30T14:09:23.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:23 smithi138 ceph-mon[33397]: osdmap e547: 8 total, 8 up, 8 in 2024-03-30T14:09:23.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:23 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:23.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:23 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:23.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:23 smithi102 ceph-mon[29392]: osdmap e547: 8 total, 8 up, 8 in 2024-03-30T14:09:23.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:23 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:23.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:23 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:23.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:23 smithi102 ceph-mon[37717]: osdmap e547: 8 total, 8 up, 8 in 2024-03-30T14:09:23.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:23 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:23.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:23 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:24 smithi138 ceph-mon[33397]: pgmap v837: 268 pgs: 32 unknown, 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:24 smithi138 ceph-mon[33397]: osdmap e548: 8 total, 8 up, 8 in 2024-03-30T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:24 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[29392]: pgmap v837: 268 pgs: 32 unknown, 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[29392]: osdmap e548: 8 total, 8 up, 8 in 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[37717]: pgmap v837: 268 pgs: 32 unknown, 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[37717]: osdmap e548: 8 total, 8 up, 8 in 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:24.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:24 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:25 smithi138 ceph-mon[33397]: osdmap e549: 8 total, 8 up, 8 in 2024-03-30T14:09:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:25.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:25 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:25.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[29392]: osdmap e549: 8 total, 8 up, 8 in 2024-03-30T14:09:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:25.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:25.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:25.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[37717]: osdmap e549: 8 total, 8 up, 8 in 2024-03-30T14:09:25.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:25.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:25 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:26.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:26 smithi138 ceph-mon[33397]: pgmap v840: 268 pgs: 32 unknown, 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:26.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:26 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:26.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:26.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:26 smithi138 ceph-mon[33397]: osdmap e550: 8 total, 8 up, 8 in 2024-03-30T14:09:26.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:26.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:26.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:09:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:09:26.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[37717]: pgmap v840: 268 pgs: 32 unknown, 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:26.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:26.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:26.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[37717]: osdmap e550: 8 total, 8 up, 8 in 2024-03-30T14:09:26.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:26.336 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:26.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[29392]: pgmap v840: 268 pgs: 32 unknown, 32 creating+peering, 204 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:26.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:26.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-107", "overlaypool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:26.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[29392]: osdmap e550: 8 total, 8 up, 8 in 2024-03-30T14:09:26.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:26.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:26.684 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:09:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:09:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]': finished 2024-03-30T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: osdmap e551: 8 total, 8 up, 8 in 2024-03-30T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]: dispatch 2024-03-30T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]: dispatch 2024-03-30T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]: dispatch 2024-03-30T14:09:27.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]: dispatch 2024-03-30T14:09:27.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]': finished 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: osdmap e551: 8 total, 8 up, 8 in 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]: dispatch 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]: dispatch 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]: dispatch 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]: dispatch 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-107-cache", "mode": "writeback"}]': finished 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: osdmap e551: 8 total, 8 up, 8 in 2024-03-30T14:09:27.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]: dispatch 2024-03-30T14:09:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]: dispatch 2024-03-30T14:09:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]: dispatch 2024-03-30T14:09:27.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]: dispatch 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: pgmap v843: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]': finished 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]': finished 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: osdmap e552: 8 total, 8 up, 8 in 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: pgmap v843: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]': finished 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]': finished 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: osdmap e552: 8 total, 8 up, 8 in 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: pgmap v843: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-107"}]': finished 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.e", "id": [7, 2]}]': finished 2024-03-30T14:09:28.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: osdmap e552: 8 total, 8 up, 8 in 2024-03-30T14:09:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3705591418' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]: dispatch 2024-03-30T14:09:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:29 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:29 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:29 smithi138 ceph-mon[33397]: osdmap e553: 8 total, 8 up, 8 in 2024-03-30T14:09:29.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:29 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:29.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:29 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:29.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:29 smithi102 ceph-mon[29392]: osdmap e553: 8 total, 8 up, 8 in 2024-03-30T14:09:29.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:29 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:29.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:29 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-107", "tierpool": "test-rados-api-smithi102-75117-107-cache"}]': finished 2024-03-30T14:09:29.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:29 smithi102 ceph-mon[37717]: osdmap e553: 8 total, 8 up, 8 in 2024-03-30T14:09:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:30 smithi138 ceph-mon[33397]: pgmap v846: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:09:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:30 smithi138 ceph-mon[33397]: osdmap e554: 8 total, 8 up, 8 in 2024-03-30T14:09:30.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:30 smithi102 ceph-mon[29392]: pgmap v846: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:09:30.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:30 smithi102 ceph-mon[29392]: osdmap e554: 8 total, 8 up, 8 in 2024-03-30T14:09:30.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:30 smithi102 ceph-mon[37717]: pgmap v846: 268 pgs: 268 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:09:30.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:30 smithi102 ceph-mon[37717]: osdmap e554: 8 total, 8 up, 8 in 2024-03-30T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:31 smithi138 ceph-mon[33397]: osdmap e555: 8 total, 8 up, 8 in 2024-03-30T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:31 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[29392]: osdmap e555: 8 total, 8 up, 8 in 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[37717]: osdmap e555: 8 total, 8 up, 8 in 2024-03-30T14:09:31.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:31.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:31 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:32 smithi138 ceph-mon[33397]: pgmap v849: 204 pgs: 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:32 smithi138 ceph-mon[33397]: osdmap e556: 8 total, 8 up, 8 in 2024-03-30T14:09:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:32.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:32 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:32.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[29392]: pgmap v849: 204 pgs: 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[29392]: osdmap e556: 8 total, 8 up, 8 in 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[37717]: pgmap v849: 204 pgs: 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[37717]: osdmap e556: 8 total, 8 up, 8 in 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:32.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:32 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:33.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:33 smithi138 ceph-mon[33397]: osdmap e557: 8 total, 8 up, 8 in 2024-03-30T14:09:33.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:33 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:33.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:33 smithi138 ceph-mon[33397]: osdmap e558: 8 total, 8 up, 8 in 2024-03-30T14:09:33.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:33 smithi102 ceph-mon[29392]: osdmap e557: 8 total, 8 up, 8 in 2024-03-30T14:09:33.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:33 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:33.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:33 smithi102 ceph-mon[29392]: osdmap e558: 8 total, 8 up, 8 in 2024-03-30T14:09:33.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:33 smithi102 ceph-mon[37717]: osdmap e557: 8 total, 8 up, 8 in 2024-03-30T14:09:33.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:33 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:33.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:33 smithi102 ceph-mon[37717]: osdmap e558: 8 total, 8 up, 8 in 2024-03-30T14:09:34.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:34 smithi138 ceph-mon[33397]: pgmap v852: 204 pgs: 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:34.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:34 smithi138 ceph-mon[33397]: osdmap e559: 8 total, 8 up, 8 in 2024-03-30T14:09:34.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:34.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:34.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[29392]: pgmap v852: 204 pgs: 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:34.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[29392]: osdmap e559: 8 total, 8 up, 8 in 2024-03-30T14:09:34.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:34.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:34.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[37717]: pgmap v852: 204 pgs: 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:34.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[37717]: osdmap e559: 8 total, 8 up, 8 in 2024-03-30T14:09:34.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:34.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:36.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:09:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:09:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:36 smithi138 ceph-mon[33397]: pgmap v855: 244 pgs: 40 unknown, 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:36 smithi138 ceph-mon[33397]: osdmap e560: 8 total, 8 up, 8 in 2024-03-30T14:09:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:36.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:36.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:36 smithi138 ceph-mon[33397]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[37717]: pgmap v855: 244 pgs: 40 unknown, 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[37717]: osdmap e560: 8 total, 8 up, 8 in 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[37717]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[29392]: pgmap v855: 244 pgs: 40 unknown, 204 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:36.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[29392]: osdmap e560: 8 total, 8 up, 8 in 2024-03-30T14:09:36.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:36.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:36.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-mon[29392]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:36.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:09:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:09:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:09:37.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:37.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:37.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: osdmap e561: 8 total, 8 up, 8 in 2024-03-30T14:09:37.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:37.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:37.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:37.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: osdmap e562: 8 total, 8 up, 8 in 2024-03-30T14:09:37.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:37.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: osdmap e561: 8 total, 8 up, 8 in 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: osdmap e562: 8 total, 8 up, 8 in 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:37.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: osdmap e561: 8 total, 8 up, 8 in 2024-03-30T14:09:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-109", "overlaypool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: osdmap e562: 8 total, 8 up, 8 in 2024-03-30T14:09:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:37.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:09:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:38 smithi138 ceph-mon[33397]: pgmap v858: 244 pgs: 244 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:38 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]': finished 2024-03-30T14:09:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:38 smithi138 ceph-mon[33397]: osdmap e563: 8 total, 8 up, 8 in 2024-03-30T14:09:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:38.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[29392]: pgmap v858: 244 pgs: 244 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]': finished 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[29392]: osdmap e563: 8 total, 8 up, 8 in 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[37717]: pgmap v858: 244 pgs: 244 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-109-cache", "mode": "writeback"}]': finished 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[37717]: osdmap e563: 8 total, 8 up, 8 in 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:09:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:40 smithi138 ceph-mon[33397]: pgmap v861: 244 pgs: 244 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:09:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:40 smithi138 ceph-mon[33397]: osdmap e564: 8 total, 8 up, 8 in 2024-03-30T14:09:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:40 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:40.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[29392]: pgmap v861: 244 pgs: 244 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:40.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[29392]: osdmap e564: 8 total, 8 up, 8 in 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[37717]: pgmap v861: 244 pgs: 244 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[37717]: osdmap e564: 8 total, 8 up, 8 in 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:40 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:09:41.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:09:41.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:41 smithi138 ceph-mon[33397]: osdmap e565: 8 total, 8 up, 8 in 2024-03-30T14:09:41.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:41.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:41.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:41 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:41.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:41 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:09:41.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:09:41.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[29392]: osdmap e565: 8 total, 8 up, 8 in 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[37717]: osdmap e565: 8 total, 8 up, 8 in 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:09:41.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:41 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:09:42.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:42 smithi138 ceph-mon[33397]: pgmap v864: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:42.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:42 smithi138 ceph-mon[33397]: osdmap e566: 8 total, 8 up, 8 in 2024-03-30T14:09:42.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T14:09:42.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T14:09:42.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:42 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:42.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:42 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-30T14:09:42.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:42 smithi138 ceph-mon[33397]: osdmap e567: 8 total, 8 up, 8 in 2024-03-30T14:09:42.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[29392]: pgmap v864: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:42.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[29392]: osdmap e566: 8 total, 8 up, 8 in 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[29392]: osdmap e567: 8 total, 8 up, 8 in 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[37717]: pgmap v864: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[37717]: osdmap e566: 8 total, 8 up, 8 in 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-30T14:09:42.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:42 smithi102 ceph-mon[37717]: osdmap e567: 8 total, 8 up, 8 in 2024-03-30T14:09:43.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:43.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:43 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:43.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:43 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:43.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:43.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:43.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:43.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:43.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:43.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:43.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:43.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:09:43.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:44 smithi138 ceph-mon[33397]: pgmap v867: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:44 smithi138 ceph-mon[33397]: osdmap e568: 8 total, 8 up, 8 in 2024-03-30T14:09:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:44 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:44.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[29392]: pgmap v867: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[29392]: osdmap e568: 8 total, 8 up, 8 in 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[37717]: pgmap v867: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[37717]: osdmap e568: 8 total, 8 up, 8 in 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:44.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:44 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]: dispatch 2024-03-30T14:09:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:45 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:45.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:45 smithi138 ceph-mon[33397]: osdmap e569: 8 total, 8 up, 8 in 2024-03-30T14:09:45.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:45 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:45.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:45 smithi102 ceph-mon[37717]: osdmap e569: 8 total, 8 up, 8 in 2024-03-30T14:09:45.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:45 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-109", "tierpool": "test-rados-api-smithi102-75117-109-cache"}]': finished 2024-03-30T14:09:45.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:45 smithi102 ceph-mon[29392]: osdmap e569: 8 total, 8 up, 8 in 2024-03-30T14:09:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:46 smithi138 ceph-mon[33397]: pgmap v870: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:46 smithi138 ceph-mon[33397]: osdmap e570: 8 total, 8 up, 8 in 2024-03-30T14:09:46.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:09:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:09:46.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:46 smithi102 ceph-mon[29392]: pgmap v870: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:46.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:46 smithi102 ceph-mon[29392]: osdmap e570: 8 total, 8 up, 8 in 2024-03-30T14:09:46.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:46 smithi102 ceph-mon[37717]: pgmap v870: 244 pgs: 244 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:46.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:46 smithi102 ceph-mon[37717]: osdmap e570: 8 total, 8 up, 8 in 2024-03-30T14:09:46.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:09:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:09:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:09:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:47 smithi138 ceph-mon[33397]: osdmap e571: 8 total, 8 up, 8 in 2024-03-30T14:09:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:47 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:47.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:47 smithi138 ceph-mon[33397]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:47.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[29392]: osdmap e571: 8 total, 8 up, 8 in 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[29392]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[37717]: osdmap e571: 8 total, 8 up, 8 in 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:47.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:47 smithi102 ceph-mon[37717]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:48 smithi138 ceph-mon[33397]: pgmap v873: 204 pgs: 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:48 smithi138 ceph-mon[33397]: osdmap e572: 8 total, 8 up, 8 in 2024-03-30T14:09:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:48 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[29392]: pgmap v873: 204 pgs: 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[29392]: osdmap e572: 8 total, 8 up, 8 in 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[37717]: pgmap v873: 204 pgs: 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[37717]: osdmap e572: 8 total, 8 up, 8 in 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2879954852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:48 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]: dispatch 2024-03-30T14:09:49.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:49 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:49.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:49 smithi138 ceph-mon[33397]: osdmap e573: 8 total, 8 up, 8 in 2024-03-30T14:09:49.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:49 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:49.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:49 smithi102 ceph-mon[37717]: osdmap e573: 8 total, 8 up, 8 in 2024-03-30T14:09:49.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:49 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-109"}]': finished 2024-03-30T14:09:49.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:49 smithi102 ceph-mon[29392]: osdmap e573: 8 total, 8 up, 8 in 2024-03-30T14:09:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:50 smithi138 ceph-mon[33397]: pgmap v876: 204 pgs: 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:50 smithi138 ceph-mon[33397]: osdmap e574: 8 total, 8 up, 8 in 2024-03-30T14:09:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:50 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:50.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[29392]: pgmap v876: 204 pgs: 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:50.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[29392]: osdmap e574: 8 total, 8 up, 8 in 2024-03-30T14:09:50.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:50.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:50.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[37717]: pgmap v876: 204 pgs: 204 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:50.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[37717]: osdmap e574: 8 total, 8 up, 8 in 2024-03-30T14:09:50.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:50.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:50 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:51.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:51.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:51 smithi138 ceph-mon[33397]: osdmap e575: 8 total, 8 up, 8 in 2024-03-30T14:09:51.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:51.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:51 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:51.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:51.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[29392]: osdmap e575: 8 total, 8 up, 8 in 2024-03-30T14:09:51.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:51.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:51.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:51.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[37717]: osdmap e575: 8 total, 8 up, 8 in 2024-03-30T14:09:51.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2331606032' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:51.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:51 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]: dispatch 2024-03-30T14:09:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:52 smithi138 ceph-mon[33397]: pgmap v879: 196 pgs: 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:09:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:52 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:52 smithi138 ceph-mon[33397]: osdmap e576: 8 total, 8 up, 8 in 2024-03-30T14:09:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:52.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:52 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:52.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[29392]: pgmap v879: 196 pgs: 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:09:52.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[29392]: osdmap e576: 8 total, 8 up, 8 in 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[37717]: pgmap v879: 196 pgs: 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi102-75117-104"}]': finished 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[37717]: osdmap e576: 8 total, 8 up, 8 in 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:09:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:52 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:53.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:53 smithi102 ceph-mon[29392]: osdmap e577: 8 total, 8 up, 8 in 2024-03-30T14:09:53.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:53.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:53.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:53 smithi102 ceph-mon[37717]: osdmap e577: 8 total, 8 up, 8 in 2024-03-30T14:09:53.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:53.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi102-75117-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:09:53.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:53 smithi138 ceph-mon[33397]: osdmap e577: 8 total, 8 up, 8 in 2024-03-30T14:09:53.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:54.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:54 smithi138 ceph-mon[33397]: pgmap v882: 196 pgs: 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:09:54.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:54 smithi138 ceph-mon[33397]: osdmap e578: 8 total, 8 up, 8 in 2024-03-30T14:09:54.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:54 smithi102 ceph-mon[29392]: pgmap v882: 196 pgs: 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:09:54.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:54 smithi102 ceph-mon[29392]: osdmap e578: 8 total, 8 up, 8 in 2024-03-30T14:09:54.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:54 smithi102 ceph-mon[37717]: pgmap v882: 196 pgs: 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:09:54.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:54 smithi102 ceph-mon[37717]: osdmap e578: 8 total, 8 up, 8 in 2024-03-30T14:09:55.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:09:55.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:55 smithi138 ceph-mon[33397]: osdmap e579: 8 total, 8 up, 8 in 2024-03-30T14:09:55.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:09:55.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:55 smithi102 ceph-mon[29392]: osdmap e579: 8 total, 8 up, 8 in 2024-03-30T14:09:55.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi102-75117-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:09:55.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:55 smithi102 ceph-mon[37717]: osdmap e579: 8 total, 8 up, 8 in 2024-03-30T14:09:56.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:09:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:09:56.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:56 smithi102 ceph-mon[29392]: pgmap v885: 204 pgs: 8 unknown, 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:56.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:56 smithi102 ceph-mon[29392]: osdmap e580: 8 total, 8 up, 8 in 2024-03-30T14:09:56.505 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:56.505 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:56 smithi102 ceph-mon[37717]: pgmap v885: 204 pgs: 8 unknown, 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:56.505 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:56 smithi102 ceph-mon[37717]: osdmap e580: 8 total, 8 up, 8 in 2024-03-30T14:09:56.505 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:56 smithi138 ceph-mon[33397]: pgmap v885: 204 pgs: 8 unknown, 196 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:09:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:56 smithi138 ceph-mon[33397]: osdmap e580: 8 total, 8 up, 8 in 2024-03-30T14:09:56.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:09:56.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:09:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:09:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:09:57.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:57 smithi138 ceph-mon[33397]: osdmap e581: 8 total, 8 up, 8 in 2024-03-30T14:09:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:09:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:57 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:57.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:57.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:57.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[29392]: osdmap e581: 8 total, 8 up, 8 in 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[37717]: osdmap e581: 8 total, 8 up, 8 in 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:09:57.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:09:58.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:58 smithi138 ceph-mon[33397]: pgmap v888: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:58.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:09:58.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:58 smithi138 ceph-mon[33397]: osdmap e582: 8 total, 8 up, 8 in 2024-03-30T14:09:58.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-112"}]: dispatch 2024-03-30T14:09:58.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[37717]: pgmap v888: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:58.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:09:58.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[37717]: osdmap e582: 8 total, 8 up, 8 in 2024-03-30T14:09:58.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-112"}]: dispatch 2024-03-30T14:09:58.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[29392]: pgmap v888: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:09:58.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:09:58.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[29392]: osdmap e582: 8 total, 8 up, 8 in 2024-03-30T14:09:58.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-112"}]: dispatch 2024-03-30T14:09:59.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-112"}]': finished 2024-03-30T14:09:59.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:59 smithi138 ceph-mon[33397]: osdmap e583: 8 total, 8 up, 8 in 2024-03-30T14:09:59.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:09:59 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:59.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-112"}]': finished 2024-03-30T14:09:59.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:59 smithi102 ceph-mon[29392]: osdmap e583: 8 total, 8 up, 8 in 2024-03-30T14:09:59.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:09:59 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:09:59.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-112"}]': finished 2024-03-30T14:09:59.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:59 smithi102 ceph-mon[37717]: osdmap e583: 8 total, 8 up, 8 in 2024-03-30T14:09:59.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:09:59 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: pgmap v891: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: osdmap e584: 8 total, 8 up, 8 in 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112"}]: dispatch 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: application not enabled on pool 'test-rados-api-smithi102-75117-111' 2024-03-30T14:10:00.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:00 smithi138 ceph-mon[33397]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:10:00.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: pgmap v891: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: osdmap e584: 8 total, 8 up, 8 in 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112"}]: dispatch 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: application not enabled on pool 'test-rados-api-smithi102-75117-111' 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[29392]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: pgmap v891: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:00.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: osdmap e584: 8 total, 8 up, 8 in 2024-03-30T14:10:00.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112"}]: dispatch 2024-03-30T14:10:00.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-30T14:10:00.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T14:10:00.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:10:00.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:10:00.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: application not enabled on pool 'test-rados-api-smithi102-75117-111' 2024-03-30T14:10:00.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:00 smithi102 ceph-mon[37717]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:10:01.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112"}]': finished 2024-03-30T14:10:01.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:01 smithi138 ceph-mon[33397]: osdmap e585: 8 total, 8 up, 8 in 2024-03-30T14:10:01.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112"}]': finished 2024-03-30T14:10:01.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:01 smithi102 ceph-mon[37717]: osdmap e585: 8 total, 8 up, 8 in 2024-03-30T14:10:01.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-112"}]': finished 2024-03-30T14:10:01.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:01 smithi102 ceph-mon[29392]: osdmap e585: 8 total, 8 up, 8 in 2024-03-30T14:10:02.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:02 smithi138 ceph-mon[33397]: pgmap v894: 236 pgs: 236 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:10:02.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:02 smithi138 ceph-mon[33397]: osdmap e586: 8 total, 8 up, 8 in 2024-03-30T14:10:02.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:02 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:02.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:02 smithi102 ceph-mon[37717]: pgmap v894: 236 pgs: 236 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:10:02.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:02 smithi102 ceph-mon[37717]: osdmap e586: 8 total, 8 up, 8 in 2024-03-30T14:10:02.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:02 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:02.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:02 smithi102 ceph-mon[29392]: pgmap v894: 236 pgs: 236 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:10:02.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:02 smithi102 ceph-mon[29392]: osdmap e586: 8 total, 8 up, 8 in 2024-03-30T14:10:02.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:02 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:03.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:03 smithi138 ceph-mon[33397]: osdmap e587: 8 total, 8 up, 8 in 2024-03-30T14:10:03.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:03.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:03 smithi102 ceph-mon[29392]: osdmap e587: 8 total, 8 up, 8 in 2024-03-30T14:10:03.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:03.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:03 smithi102 ceph-mon[37717]: osdmap e587: 8 total, 8 up, 8 in 2024-03-30T14:10:03.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:04.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:04 smithi138 ceph-mon[33397]: pgmap v897: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:04.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:04.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:04 smithi138 ceph-mon[33397]: osdmap e588: 8 total, 8 up, 8 in 2024-03-30T14:10:04.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:04.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[29392]: pgmap v897: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:04.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:04.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[29392]: osdmap e588: 8 total, 8 up, 8 in 2024-03-30T14:10:04.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:04.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[37717]: pgmap v897: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:04.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:04.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[37717]: osdmap e588: 8 total, 8 up, 8 in 2024-03-30T14:10:04.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:05 smithi138 ceph-mon[33397]: osdmap e589: 8 total, 8 up, 8 in 2024-03-30T14:10:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-114"}]: dispatch 2024-03-30T14:10:05.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-114"}]': finished 2024-03-30T14:10:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[29392]: osdmap e589: 8 total, 8 up, 8 in 2024-03-30T14:10:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-114"}]: dispatch 2024-03-30T14:10:05.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-114"}]': finished 2024-03-30T14:10:05.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:05.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[37717]: osdmap e589: 8 total, 8 up, 8 in 2024-03-30T14:10:05.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-114"}]: dispatch 2024-03-30T14:10:05.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-114"}]': finished 2024-03-30T14:10:06.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:10:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:10:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:06 smithi138 ceph-mon[33397]: pgmap v900: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:06 smithi138 ceph-mon[33397]: osdmap e590: 8 total, 8 up, 8 in 2024-03-30T14:10:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-114", "mode": "writeback"}]: dispatch 2024-03-30T14:10:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:06 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-114", "mode": "writeback"}]': finished 2024-03-30T14:10:06.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:06 smithi138 ceph-mon[33397]: osdmap e591: 8 total, 8 up, 8 in 2024-03-30T14:10:06.684 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:10:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[29392]: pgmap v900: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[29392]: osdmap e590: 8 total, 8 up, 8 in 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-114", "mode": "writeback"}]: dispatch 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-114", "mode": "writeback"}]': finished 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[29392]: osdmap e591: 8 total, 8 up, 8 in 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[37717]: pgmap v900: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[37717]: osdmap e590: 8 total, 8 up, 8 in 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-114", "mode": "writeback"}]: dispatch 2024-03-30T14:10:06.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:06.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-114", "mode": "writeback"}]': finished 2024-03-30T14:10:06.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:06 smithi102 ceph-mon[37717]: osdmap e591: 8 total, 8 up, 8 in 2024-03-30T14:10:07.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:07.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:07.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:07 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:07.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:07.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:07.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:07 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:07.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:07.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:07.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:07 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:08.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:08 smithi138 ceph-mon[33397]: pgmap v903: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:08.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:08.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:08 smithi138 ceph-mon[33397]: osdmap e592: 8 total, 8 up, 8 in 2024-03-30T14:10:08.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114"}]: dispatch 2024-03-30T14:10:08.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[37717]: pgmap v903: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:08.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:08.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[37717]: osdmap e592: 8 total, 8 up, 8 in 2024-03-30T14:10:08.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114"}]: dispatch 2024-03-30T14:10:08.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[29392]: pgmap v903: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:08.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:08.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[29392]: osdmap e592: 8 total, 8 up, 8 in 2024-03-30T14:10:08.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114"}]: dispatch 2024-03-30T14:10:09.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:09 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:09.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114"}]': finished 2024-03-30T14:10:09.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:09 smithi138 ceph-mon[33397]: osdmap e593: 8 total, 8 up, 8 in 2024-03-30T14:10:09.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:09 smithi138 ceph-mon[33397]: pgmap v906: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:09.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:09.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114"}]': finished 2024-03-30T14:10:09.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[29392]: osdmap e593: 8 total, 8 up, 8 in 2024-03-30T14:10:09.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[29392]: pgmap v906: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:09.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:09.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-114"}]': finished 2024-03-30T14:10:09.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[37717]: osdmap e593: 8 total, 8 up, 8 in 2024-03-30T14:10:09.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:09 smithi102 ceph-mon[37717]: pgmap v906: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:10.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:10 smithi138 ceph-mon[33397]: osdmap e594: 8 total, 8 up, 8 in 2024-03-30T14:10:10.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:10 smithi102 ceph-mon[29392]: osdmap e594: 8 total, 8 up, 8 in 2024-03-30T14:10:10.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:10 smithi102 ceph-mon[37717]: osdmap e594: 8 total, 8 up, 8 in 2024-03-30T14:10:11.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:11 smithi138 ceph-mon[33397]: osdmap e595: 8 total, 8 up, 8 in 2024-03-30T14:10:11.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:11.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:11 smithi138 ceph-mon[33397]: pgmap v909: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:11.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:11 smithi102 ceph-mon[29392]: osdmap e595: 8 total, 8 up, 8 in 2024-03-30T14:10:11.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:11.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:11 smithi102 ceph-mon[29392]: pgmap v909: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:11.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:11 smithi102 ceph-mon[37717]: osdmap e595: 8 total, 8 up, 8 in 2024-03-30T14:10:11.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:11.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:11 smithi102 ceph-mon[37717]: pgmap v909: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:12.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:12.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:12 smithi138 ceph-mon[33397]: osdmap e596: 8 total, 8 up, 8 in 2024-03-30T14:10:12.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:12 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:12.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:10:12.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:12.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:12.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[29392]: osdmap e596: 8 total, 8 up, 8 in 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[37717]: osdmap e596: 8 total, 8 up, 8 in 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:10:12.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:13 smithi138 ceph-mon[33397]: osdmap e597: 8 total, 8 up, 8 in 2024-03-30T14:10:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:13 smithi138 ceph-mon[33397]: pgmap v912: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:13.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:13 smithi102 ceph-mon[29392]: osdmap e597: 8 total, 8 up, 8 in 2024-03-30T14:10:13.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:13.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:13 smithi102 ceph-mon[29392]: pgmap v912: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:13.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:13 smithi102 ceph-mon[37717]: osdmap e597: 8 total, 8 up, 8 in 2024-03-30T14:10:13.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:13.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:13 smithi102 ceph-mon[37717]: pgmap v912: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:14.580 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:14.580 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:14 smithi138 ceph-mon[33397]: osdmap e598: 8 total, 8 up, 8 in 2024-03-30T14:10:14.580 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-116"}]: dispatch 2024-03-30T14:10:14.589 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:14.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:14 smithi102 ceph-mon[37717]: osdmap e598: 8 total, 8 up, 8 in 2024-03-30T14:10:14.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-116"}]: dispatch 2024-03-30T14:10:14.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:14.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:14 smithi102 ceph-mon[29392]: osdmap e598: 8 total, 8 up, 8 in 2024-03-30T14:10:14.590 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-116"}]: dispatch 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-116"}]': finished 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: osdmap e599: 8 total, 8 up, 8 in 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-116", "mode": "writeback"}]: dispatch 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: pgmap v915: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-116", "mode": "writeback"}]': finished 2024-03-30T14:10:15.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-mon[33397]: osdmap e600: 8 total, 8 up, 8 in 2024-03-30T14:10:15.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-116"}]': finished 2024-03-30T14:10:15.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: osdmap e599: 8 total, 8 up, 8 in 2024-03-30T14:10:15.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-116", "mode": "writeback"}]: dispatch 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: pgmap v915: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-116", "mode": "writeback"}]': finished 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[37717]: osdmap e600: 8 total, 8 up, 8 in 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-116"}]': finished 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: osdmap e599: 8 total, 8 up, 8 in 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-116", "mode": "writeback"}]: dispatch 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: pgmap v915: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:15.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-116", "mode": "writeback"}]': finished 2024-03-30T14:10:15.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:15 smithi102 ceph-mon[29392]: osdmap e600: 8 total, 8 up, 8 in 2024-03-30T14:10:16.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:10:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:10:16.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "313.0"}]: dispatch 2024-03-30T14:10:16.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:16 smithi138 ceph-mon[33397]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "313.0"}]: dispatch 2024-03-30T14:10:16.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:16 smithi138 ceph-mon[33397]: 313.0 scrub starts 2024-03-30T14:10:16.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:16 smithi138 ceph-mon[33397]: 313.0 scrub ok 2024-03-30T14:10:16.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:16.684 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:10:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "313.0"}]: dispatch 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[37717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "313.0"}]: dispatch 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[37717]: 313.0 scrub starts 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[37717]: 313.0 scrub ok 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "313.0"}]: dispatch 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[29392]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "313.0"}]: dispatch 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[29392]: 313.0 scrub starts 2024-03-30T14:10:16.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[29392]: 313.0 scrub ok 2024-03-30T14:10:16.686 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:17 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:17 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:17 smithi138 ceph-mon[33397]: pgmap v917: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T14:10:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:17 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:17 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:10:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:17 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:10:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:17 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[29392]: pgmap v917: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T14:10:17.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:17.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:10:17.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:10:17.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:17.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[37717]: pgmap v917: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T14:10:17.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:17.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:10:17.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:10:17.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:17 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:19.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:19 smithi138 ceph-mon[33397]: pgmap v918: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:10:19.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:19 smithi102 ceph-mon[29392]: pgmap v918: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:10:19.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:19 smithi102 ceph-mon[37717]: pgmap v918: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:10:21.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:21 smithi138 ceph-mon[33397]: pgmap v919: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1003 B/s wr, 3 op/s 2024-03-30T14:10:21.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:21 smithi102 ceph-mon[29392]: pgmap v919: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1003 B/s wr, 3 op/s 2024-03-30T14:10:21.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:21 smithi102 ceph-mon[37717]: pgmap v919: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1003 B/s wr, 3 op/s 2024-03-30T14:10:23.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:23 smithi138 ceph-mon[33397]: pgmap v920: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 880 B/s wr, 2 op/s 2024-03-30T14:10:23.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:23 smithi102 ceph-mon[29392]: pgmap v920: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 880 B/s wr, 2 op/s 2024-03-30T14:10:23.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:23 smithi102 ceph-mon[37717]: pgmap v920: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 880 B/s wr, 2 op/s 2024-03-30T14:10:25.920 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:25 smithi138 ceph-mon[33397]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-03-30T14:10:25.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:25 smithi102 ceph-mon[37717]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-03-30T14:10:25.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:25 smithi102 ceph-mon[29392]: pgmap v921: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-03-30T14:10:26.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:10:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:10:26.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:26.835 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:26.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:10:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:10:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:10:26.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:27.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:27 smithi138 ceph-mon[33397]: pgmap v922: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 459 B/s wr, 1 op/s 2024-03-30T14:10:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]: dispatch 2024-03-30T14:10:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]: dispatch 2024-03-30T14:10:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:27.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:27.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[29392]: pgmap v922: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 459 B/s wr, 1 op/s 2024-03-30T14:10:27.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]: dispatch 2024-03-30T14:10:27.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]: dispatch 2024-03-30T14:10:27.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:27.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:27.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[37717]: pgmap v922: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 459 B/s wr, 1 op/s 2024-03-30T14:10:27.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]: dispatch 2024-03-30T14:10:27.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]: dispatch 2024-03-30T14:10:27.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:27.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:28.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:28 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]': finished 2024-03-30T14:10:28.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:28 smithi138 ceph-mon[33397]: osdmap e601: 8 total, 8 up, 8 in 2024-03-30T14:10:28.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:28 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]': finished 2024-03-30T14:10:28.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:28 smithi102 ceph-mon[29392]: osdmap e601: 8 total, 8 up, 8 in 2024-03-30T14:10:28.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:28 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.17", "id": [0, 1]}]': finished 2024-03-30T14:10:28.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:28 smithi102 ceph-mon[37717]: osdmap e601: 8 total, 8 up, 8 in 2024-03-30T14:10:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:29 smithi138 ceph-mon[33397]: pgmap v924: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:29.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:29 smithi138 ceph-mon[33397]: osdmap e602: 8 total, 8 up, 8 in 2024-03-30T14:10:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:29 smithi102 ceph-mon[37717]: pgmap v924: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:29 smithi102 ceph-mon[37717]: osdmap e602: 8 total, 8 up, 8 in 2024-03-30T14:10:29.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:29 smithi102 ceph-mon[29392]: pgmap v924: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:29.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:29 smithi102 ceph-mon[29392]: osdmap e602: 8 total, 8 up, 8 in 2024-03-30T14:10:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:31 smithi138 ceph-mon[33397]: pgmap v926: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:31.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:31 smithi138 ceph-mon[33397]: osdmap e603: 8 total, 8 up, 8 in 2024-03-30T14:10:31.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:31 smithi102 ceph-mon[29392]: pgmap v926: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:31 smithi102 ceph-mon[29392]: osdmap e603: 8 total, 8 up, 8 in 2024-03-30T14:10:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:31.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:31 smithi102 ceph-mon[37717]: pgmap v926: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:31.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:31 smithi102 ceph-mon[37717]: osdmap e603: 8 total, 8 up, 8 in 2024-03-30T14:10:31.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:32.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:32.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:32 smithi138 ceph-mon[33397]: osdmap e604: 8 total, 8 up, 8 in 2024-03-30T14:10:32.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116"}]: dispatch 2024-03-30T14:10:32.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:32 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:32.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116"}]': finished 2024-03-30T14:10:32.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:32.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[29392]: osdmap e604: 8 total, 8 up, 8 in 2024-03-30T14:10:32.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116"}]: dispatch 2024-03-30T14:10:32.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:32.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116"}]': finished 2024-03-30T14:10:32.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:32.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[37717]: osdmap e604: 8 total, 8 up, 8 in 2024-03-30T14:10:32.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116"}]: dispatch 2024-03-30T14:10:32.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:32.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-116"}]': finished 2024-03-30T14:10:33.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:33 smithi138 ceph-mon[33397]: pgmap v929: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:10:33.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:33 smithi138 ceph-mon[33397]: osdmap e605: 8 total, 8 up, 8 in 2024-03-30T14:10:33.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:33 smithi102 ceph-mon[29392]: pgmap v929: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:10:33.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:33 smithi102 ceph-mon[29392]: osdmap e605: 8 total, 8 up, 8 in 2024-03-30T14:10:33.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:33 smithi102 ceph-mon[37717]: pgmap v929: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:10:33.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:33 smithi102 ceph-mon[37717]: osdmap e605: 8 total, 8 up, 8 in 2024-03-30T14:10:34.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:34 smithi138 ceph-mon[33397]: osdmap e606: 8 total, 8 up, 8 in 2024-03-30T14:10:34.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:34 smithi102 ceph-mon[29392]: osdmap e606: 8 total, 8 up, 8 in 2024-03-30T14:10:34.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:34 smithi102 ceph-mon[37717]: osdmap e606: 8 total, 8 up, 8 in 2024-03-30T14:10:35.924 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:35 smithi138 ceph-mon[33397]: pgmap v932: 204 pgs: 204 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:35.924 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:35 smithi138 ceph-mon[33397]: osdmap e607: 8 total, 8 up, 8 in 2024-03-30T14:10:35.924 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:35.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:35 smithi102 ceph-mon[29392]: pgmap v932: 204 pgs: 204 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:35.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:35 smithi102 ceph-mon[29392]: osdmap e607: 8 total, 8 up, 8 in 2024-03-30T14:10:35.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:35.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:35 smithi102 ceph-mon[37717]: pgmap v932: 204 pgs: 204 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:35.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:35 smithi102 ceph-mon[37717]: osdmap e607: 8 total, 8 up, 8 in 2024-03-30T14:10:35.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:36.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:10:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:10:36.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:36.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:36 smithi138 ceph-mon[33397]: osdmap e608: 8 total, 8 up, 8 in 2024-03-30T14:10:36.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:36.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:10:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:10:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:10:36.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:36.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:36 smithi102 ceph-mon[29392]: osdmap e608: 8 total, 8 up, 8 in 2024-03-30T14:10:36.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:36 smithi102 ceph-mon[37717]: osdmap e608: 8 total, 8 up, 8 in 2024-03-30T14:10:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:37.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:37 smithi138 ceph-mon[33397]: pgmap v935: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-30T14:10:37.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:37 smithi138 ceph-mon[33397]: osdmap e609: 8 total, 8 up, 8 in 2024-03-30T14:10:37.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:37.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:37.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:37 smithi138 ceph-mon[33397]: osdmap e610: 8 total, 8 up, 8 in 2024-03-30T14:10:37.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-118"}]: dispatch 2024-03-30T14:10:37.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[29392]: pgmap v935: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-30T14:10:37.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[29392]: osdmap e609: 8 total, 8 up, 8 in 2024-03-30T14:10:37.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:37.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[29392]: osdmap e610: 8 total, 8 up, 8 in 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-118"}]: dispatch 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[37717]: pgmap v935: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[37717]: osdmap e609: 8 total, 8 up, 8 in 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[37717]: osdmap e610: 8 total, 8 up, 8 in 2024-03-30T14:10:37.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-118"}]: dispatch 2024-03-30T14:10:38.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-118"}]': finished 2024-03-30T14:10:38.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:38 smithi138 ceph-mon[33397]: osdmap e611: 8 total, 8 up, 8 in 2024-03-30T14:10:38.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-118", "mode": "writeback"}]: dispatch 2024-03-30T14:10:38.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-118"}]': finished 2024-03-30T14:10:38.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:38 smithi102 ceph-mon[37717]: osdmap e611: 8 total, 8 up, 8 in 2024-03-30T14:10:38.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-118", "mode": "writeback"}]: dispatch 2024-03-30T14:10:38.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-118"}]': finished 2024-03-30T14:10:38.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:38 smithi102 ceph-mon[29392]: osdmap e611: 8 total, 8 up, 8 in 2024-03-30T14:10:38.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-118", "mode": "writeback"}]: dispatch 2024-03-30T14:10:39.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:39 smithi138 ceph-mon[33397]: pgmap v939: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:10:39.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:39 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:39.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-118", "mode": "writeback"}]': finished 2024-03-30T14:10:39.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:39 smithi138 ceph-mon[33397]: osdmap e612: 8 total, 8 up, 8 in 2024-03-30T14:10:39.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[29392]: pgmap v939: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:10:39.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:39.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-118", "mode": "writeback"}]': finished 2024-03-30T14:10:39.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[29392]: osdmap e612: 8 total, 8 up, 8 in 2024-03-30T14:10:39.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[37717]: pgmap v939: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T14:10:39.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:39.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-118", "mode": "writeback"}]': finished 2024-03-30T14:10:39.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:39 smithi102 ceph-mon[37717]: osdmap e612: 8 total, 8 up, 8 in 2024-03-30T14:10:40.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:40 smithi102 ceph-mon[29392]: osdmap e613: 8 total, 8 up, 8 in 2024-03-30T14:10:40.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:40 smithi102 ceph-mon[37717]: osdmap e613: 8 total, 8 up, 8 in 2024-03-30T14:10:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:41.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:40 smithi138 ceph-mon[33397]: osdmap e613: 8 total, 8 up, 8 in 2024-03-30T14:10:41.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:41.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:41 smithi138 ceph-mon[33397]: pgmap v942: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 530 B/s wr, 2 op/s 2024-03-30T14:10:41.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:41.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:41 smithi138 ceph-mon[33397]: osdmap e614: 8 total, 8 up, 8 in 2024-03-30T14:10:41.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118"}]: dispatch 2024-03-30T14:10:41.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:41 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:41.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118"}]': finished 2024-03-30T14:10:41.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:41 smithi138 ceph-mon[33397]: osdmap e615: 8 total, 8 up, 8 in 2024-03-30T14:10:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[29392]: pgmap v942: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 530 B/s wr, 2 op/s 2024-03-30T14:10:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[29392]: osdmap e614: 8 total, 8 up, 8 in 2024-03-30T14:10:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118"}]: dispatch 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118"}]': finished 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[29392]: osdmap e615: 8 total, 8 up, 8 in 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[37717]: pgmap v942: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 530 B/s wr, 2 op/s 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[37717]: osdmap e614: 8 total, 8 up, 8 in 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118"}]: dispatch 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-118"}]': finished 2024-03-30T14:10:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:41 smithi102 ceph-mon[37717]: osdmap e615: 8 total, 8 up, 8 in 2024-03-30T14:10:43.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:43 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:43.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:43.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:43 smithi138 ceph-mon[33397]: osdmap e616: 8 total, 8 up, 8 in 2024-03-30T14:10:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:43 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:43.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:43.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:43 smithi102 ceph-mon[29392]: osdmap e616: 8 total, 8 up, 8 in 2024-03-30T14:10:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:43 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:43 smithi102 ceph-mon[37717]: osdmap e616: 8 total, 8 up, 8 in 2024-03-30T14:10:44.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:44 smithi138 ceph-mon[33397]: pgmap v945: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:10:44.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:44 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:44.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:44 smithi138 ceph-mon[33397]: osdmap e617: 8 total, 8 up, 8 in 2024-03-30T14:10:44.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[29392]: pgmap v945: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:10:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[29392]: osdmap e617: 8 total, 8 up, 8 in 2024-03-30T14:10:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[37717]: pgmap v945: 236 pgs: 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:10:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[37717]: osdmap e617: 8 total, 8 up, 8 in 2024-03-30T14:10:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:46.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:10:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:10:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:45 smithi138 ceph-mon[33397]: pgmap v948: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:45 smithi138 ceph-mon[33397]: osdmap e618: 8 total, 8 up, 8 in 2024-03-30T14:10:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:46.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[29392]: pgmap v948: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:46.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:46.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[29392]: osdmap e618: 8 total, 8 up, 8 in 2024-03-30T14:10:46.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:46.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[37717]: pgmap v948: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:46.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:46.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[37717]: osdmap e618: 8 total, 8 up, 8 in 2024-03-30T14:10:46.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:46.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:10:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:10:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[37717]: osdmap e619: 8 total, 8 up, 8 in 2024-03-30T14:10:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-120"}]: dispatch 2024-03-30T14:10:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-120"}]': finished 2024-03-30T14:10:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[37717]: osdmap e620: 8 total, 8 up, 8 in 2024-03-30T14:10:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-120", "mode": "writeback"}]: dispatch 2024-03-30T14:10:46.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:46.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[29392]: osdmap e619: 8 total, 8 up, 8 in 2024-03-30T14:10:46.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-120"}]: dispatch 2024-03-30T14:10:46.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:46.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-120"}]': finished 2024-03-30T14:10:46.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[29392]: osdmap e620: 8 total, 8 up, 8 in 2024-03-30T14:10:46.936 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-120", "mode": "writeback"}]: dispatch 2024-03-30T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:46 smithi138 ceph-mon[33397]: osdmap e619: 8 total, 8 up, 8 in 2024-03-30T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-120"}]: dispatch 2024-03-30T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-120"}]': finished 2024-03-30T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:46 smithi138 ceph-mon[33397]: osdmap e620: 8 total, 8 up, 8 in 2024-03-30T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-120", "mode": "writeback"}]: dispatch 2024-03-30T14:10:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:47 smithi138 ceph-mon[33397]: pgmap v951: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:48.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:47 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:48.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-120", "mode": "writeback"}]': finished 2024-03-30T14:10:48.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:47 smithi138 ceph-mon[33397]: osdmap e621: 8 total, 8 up, 8 in 2024-03-30T14:10:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[29392]: pgmap v951: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-120", "mode": "writeback"}]': finished 2024-03-30T14:10:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[29392]: osdmap e621: 8 total, 8 up, 8 in 2024-03-30T14:10:48.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[37717]: pgmap v951: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:10:48.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:48.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-120", "mode": "writeback"}]': finished 2024-03-30T14:10:48.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:47 smithi102 ceph-mon[37717]: osdmap e621: 8 total, 8 up, 8 in 2024-03-30T14:10:49.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:49.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:49.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:50.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:49 smithi138 ceph-mon[33397]: pgmap v954: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:49 smithi138 ceph-mon[33397]: osdmap e622: 8 total, 8 up, 8 in 2024-03-30T14:10:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120"}]: dispatch 2024-03-30T14:10:50.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[29392]: pgmap v954: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[29392]: osdmap e622: 8 total, 8 up, 8 in 2024-03-30T14:10:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120"}]: dispatch 2024-03-30T14:10:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[37717]: pgmap v954: 236 pgs: 236 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[37717]: osdmap e622: 8 total, 8 up, 8 in 2024-03-30T14:10:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120"}]: dispatch 2024-03-30T14:10:51.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:50 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:51.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120"}]': finished 2024-03-30T14:10:51.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:50 smithi138 ceph-mon[33397]: osdmap e623: 8 total, 8 up, 8 in 2024-03-30T14:10:51.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:50 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:51.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120"}]': finished 2024-03-30T14:10:51.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:50 smithi102 ceph-mon[37717]: osdmap e623: 8 total, 8 up, 8 in 2024-03-30T14:10:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:50 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-120"}]': finished 2024-03-30T14:10:51.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:50 smithi102 ceph-mon[29392]: osdmap e623: 8 total, 8 up, 8 in 2024-03-30T14:10:52.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:51 smithi138 ceph-mon[33397]: pgmap v957: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T14:10:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:51 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:51 smithi138 ceph-mon[33397]: osdmap e624: 8 total, 8 up, 8 in 2024-03-30T14:10:52.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:51 smithi102 ceph-mon[37717]: pgmap v957: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T14:10:52.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:51 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:52.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:51 smithi102 ceph-mon[37717]: osdmap e624: 8 total, 8 up, 8 in 2024-03-30T14:10:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:51 smithi102 ceph-mon[29392]: pgmap v957: 236 pgs: 236 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T14:10:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:51 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:51 smithi102 ceph-mon[29392]: osdmap e624: 8 total, 8 up, 8 in 2024-03-30T14:10:53.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:52 smithi138 ceph-mon[33397]: osdmap e625: 8 total, 8 up, 8 in 2024-03-30T14:10:53.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:53.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:52 smithi102 ceph-mon[37717]: osdmap e625: 8 total, 8 up, 8 in 2024-03-30T14:10:53.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:53.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:52 smithi102 ceph-mon[29392]: osdmap e625: 8 total, 8 up, 8 in 2024-03-30T14:10:53.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:10:54.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:53 smithi138 ceph-mon[33397]: pgmap v960: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:53 smithi138 ceph-mon[33397]: osdmap e626: 8 total, 8 up, 8 in 2024-03-30T14:10:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[29392]: pgmap v960: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[29392]: osdmap e626: 8 total, 8 up, 8 in 2024-03-30T14:10:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[37717]: pgmap v960: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:10:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[37717]: osdmap e626: 8 total, 8 up, 8 in 2024-03-30T14:10:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:10:55.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:54 smithi138 ceph-mon[33397]: osdmap e627: 8 total, 8 up, 8 in 2024-03-30T14:10:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-122"}]: dispatch 2024-03-30T14:10:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-122"}]': finished 2024-03-30T14:10:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[29392]: osdmap e627: 8 total, 8 up, 8 in 2024-03-30T14:10:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-122"}]: dispatch 2024-03-30T14:10:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-122"}]': finished 2024-03-30T14:10:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:10:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[37717]: osdmap e627: 8 total, 8 up, 8 in 2024-03-30T14:10:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-122"}]: dispatch 2024-03-30T14:10:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-122"}]': finished 2024-03-30T14:10:56.085 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[37717]: pgmap v963: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[37717]: osdmap e628: 8 total, 8 up, 8 in 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-122", "mode": "writeback"}]: dispatch 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-122", "mode": "writeback"}]': finished 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[37717]: osdmap e629: 8 total, 8 up, 8 in 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[29392]: pgmap v963: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[29392]: osdmap e628: 8 total, 8 up, 8 in 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-122", "mode": "writeback"}]: dispatch 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-122", "mode": "writeback"}]': finished 2024-03-30T14:10:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:55 smithi102 ceph-mon[29392]: osdmap e629: 8 total, 8 up, 8 in 2024-03-30T14:10:56.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:10:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:10:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:55 smithi138 ceph-mon[33397]: pgmap v963: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:10:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:55 smithi138 ceph-mon[33397]: osdmap e628: 8 total, 8 up, 8 in 2024-03-30T14:10:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-122", "mode": "writeback"}]: dispatch 2024-03-30T14:10:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:55 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:10:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-122", "mode": "writeback"}]': finished 2024-03-30T14:10:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:55 smithi138 ceph-mon[33397]: osdmap e629: 8 total, 8 up, 8 in 2024-03-30T14:10:56.792 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:10:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:10:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:10:57.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:57.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:57.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:56 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:57.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:57.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:57.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:56 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:57.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:10:57.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:10:57.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:56 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:10:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:57 smithi138 ceph-mon[33397]: pgmap v966: 236 pgs: 236 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:57 smithi138 ceph-mon[33397]: osdmap e630: 8 total, 8 up, 8 in 2024-03-30T14:10:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122"}]: dispatch 2024-03-30T14:10:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:57 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[29392]: pgmap v966: 236 pgs: 236 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[29392]: osdmap e630: 8 total, 8 up, 8 in 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122"}]: dispatch 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[37717]: pgmap v966: 236 pgs: 236 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[37717]: osdmap e630: 8 total, 8 up, 8 in 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122"}]: dispatch 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:10:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:10:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:58 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122"}]': finished 2024-03-30T14:10:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:58 smithi138 ceph-mon[33397]: osdmap e631: 8 total, 8 up, 8 in 2024-03-30T14:10:59.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:58 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:59.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122"}]': finished 2024-03-30T14:10:59.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:58 smithi102 ceph-mon[29392]: osdmap e631: 8 total, 8 up, 8 in 2024-03-30T14:10:59.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:58 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:10:59.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-122"}]': finished 2024-03-30T14:10:59.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:58 smithi102 ceph-mon[37717]: osdmap e631: 8 total, 8 up, 8 in 2024-03-30T14:11:00.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:59 smithi138 ceph-mon[33397]: pgmap v969: 236 pgs: 236 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:00.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:10:59 smithi138 ceph-mon[33397]: osdmap e632: 8 total, 8 up, 8 in 2024-03-30T14:11:00.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:59 smithi102 ceph-mon[29392]: pgmap v969: 236 pgs: 236 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:00.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:10:59 smithi102 ceph-mon[29392]: osdmap e632: 8 total, 8 up, 8 in 2024-03-30T14:11:00.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:59 smithi102 ceph-mon[37717]: pgmap v969: 236 pgs: 236 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:00.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:10:59 smithi102 ceph-mon[37717]: osdmap e632: 8 total, 8 up, 8 in 2024-03-30T14:11:01.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[29392]: osdmap e633: 8 total, 8 up, 8 in 2024-03-30T14:11:01.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:01.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:01.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[29392]: osdmap e634: 8 total, 8 up, 8 in 2024-03-30T14:11:01.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[37717]: osdmap e633: 8 total, 8 up, 8 in 2024-03-30T14:11:01.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:01.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:01.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:00 smithi102 ceph-mon[37717]: osdmap e634: 8 total, 8 up, 8 in 2024-03-30T14:11:01.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:00 smithi138 ceph-mon[33397]: osdmap e633: 8 total, 8 up, 8 in 2024-03-30T14:11:01.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:00 smithi138 ceph-mon[33397]: osdmap e634: 8 total, 8 up, 8 in 2024-03-30T14:11:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:01 smithi138 ceph-mon[33397]: pgmap v972: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:01 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:01 smithi138 ceph-mon[33397]: osdmap e635: 8 total, 8 up, 8 in 2024-03-30T14:11:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:01 smithi102 ceph-mon[29392]: pgmap v972: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:01 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:01 smithi102 ceph-mon[29392]: osdmap e635: 8 total, 8 up, 8 in 2024-03-30T14:11:02.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:01 smithi102 ceph-mon[37717]: pgmap v972: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:01 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:01 smithi102 ceph-mon[37717]: osdmap e635: 8 total, 8 up, 8 in 2024-03-30T14:11:03.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:02 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:03.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:02 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:03.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:02 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:03 smithi138 ceph-mon[33397]: pgmap v975: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:03 smithi138 ceph-mon[33397]: osdmap e636: 8 total, 8 up, 8 in 2024-03-30T14:11:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-124"}]: dispatch 2024-03-30T14:11:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[29392]: pgmap v975: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[29392]: osdmap e636: 8 total, 8 up, 8 in 2024-03-30T14:11:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-124"}]: dispatch 2024-03-30T14:11:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[37717]: pgmap v975: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[37717]: osdmap e636: 8 total, 8 up, 8 in 2024-03-30T14:11:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-124"}]: dispatch 2024-03-30T14:11:05.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-124"}]': finished 2024-03-30T14:11:05.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:04 smithi138 ceph-mon[33397]: osdmap e637: 8 total, 8 up, 8 in 2024-03-30T14:11:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-124", "mode": "writeback"}]: dispatch 2024-03-30T14:11:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:04 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-124", "mode": "writeback"}]': finished 2024-03-30T14:11:05.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:04 smithi138 ceph-mon[33397]: osdmap e638: 8 total, 8 up, 8 in 2024-03-30T14:11:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-124"}]': finished 2024-03-30T14:11:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[29392]: osdmap e637: 8 total, 8 up, 8 in 2024-03-30T14:11:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-124", "mode": "writeback"}]: dispatch 2024-03-30T14:11:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-124", "mode": "writeback"}]': finished 2024-03-30T14:11:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[29392]: osdmap e638: 8 total, 8 up, 8 in 2024-03-30T14:11:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-124"}]': finished 2024-03-30T14:11:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[37717]: osdmap e637: 8 total, 8 up, 8 in 2024-03-30T14:11:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-124", "mode": "writeback"}]: dispatch 2024-03-30T14:11:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-124", "mode": "writeback"}]': finished 2024-03-30T14:11:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:04 smithi102 ceph-mon[37717]: osdmap e638: 8 total, 8 up, 8 in 2024-03-30T14:11:06.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:05 smithi138 ceph-mon[33397]: pgmap v978: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:06.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:11:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:11:06.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:05 smithi102 ceph-mon[29392]: pgmap v978: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:06.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:05 smithi102 ceph-mon[37717]: pgmap v978: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:06.886 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:11:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:11:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:06 smithi138 ceph-mon[33397]: osdmap e639: 8 total, 8 up, 8 in 2024-03-30T14:11:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:06 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[37717]: osdmap e639: 8 total, 8 up, 8 in 2024-03-30T14:11:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[29392]: osdmap e639: 8 total, 8 up, 8 in 2024-03-30T14:11:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:06 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:08.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:07 smithi138 ceph-mon[33397]: pgmap v981: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:11:08.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:08.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:07 smithi138 ceph-mon[33397]: osdmap e640: 8 total, 8 up, 8 in 2024-03-30T14:11:08.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124"}]: dispatch 2024-03-30T14:11:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[29392]: pgmap v981: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:11:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[29392]: osdmap e640: 8 total, 8 up, 8 in 2024-03-30T14:11:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124"}]: dispatch 2024-03-30T14:11:08.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[37717]: pgmap v981: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:11:08.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:08.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[37717]: osdmap e640: 8 total, 8 up, 8 in 2024-03-30T14:11:08.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124"}]: dispatch 2024-03-30T14:11:09.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:08 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:09.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124"}]': finished 2024-03-30T14:11:09.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:08 smithi138 ceph-mon[33397]: osdmap e641: 8 total, 8 up, 8 in 2024-03-30T14:11:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:08 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124"}]': finished 2024-03-30T14:11:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:08 smithi102 ceph-mon[37717]: osdmap e641: 8 total, 8 up, 8 in 2024-03-30T14:11:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:08 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-124"}]': finished 2024-03-30T14:11:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:08 smithi102 ceph-mon[29392]: osdmap e641: 8 total, 8 up, 8 in 2024-03-30T14:11:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:09 smithi138 ceph-mon[33397]: pgmap v984: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:11:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:09 smithi138 ceph-mon[33397]: osdmap e642: 8 total, 8 up, 8 in 2024-03-30T14:11:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:09 smithi102 ceph-mon[29392]: pgmap v984: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:11:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:09 smithi102 ceph-mon[29392]: osdmap e642: 8 total, 8 up, 8 in 2024-03-30T14:11:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:09 smithi102 ceph-mon[37717]: pgmap v984: 236 pgs: 236 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T14:11:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:09 smithi102 ceph-mon[37717]: osdmap e642: 8 total, 8 up, 8 in 2024-03-30T14:11:11.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:10 smithi138 ceph-mon[33397]: osdmap e643: 8 total, 8 up, 8 in 2024-03-30T14:11:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:11.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:10 smithi102 ceph-mon[29392]: osdmap e643: 8 total, 8 up, 8 in 2024-03-30T14:11:11.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:10 smithi102 ceph-mon[37717]: osdmap e643: 8 total, 8 up, 8 in 2024-03-30T14:11:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: pgmap v987: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: osdmap e644: 8 total, 8 up, 8 in 2024-03-30T14:11:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: osdmap e645: 8 total, 8 up, 8 in 2024-03-30T14:11:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-126"}]: dispatch 2024-03-30T14:11:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: pgmap v987: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: osdmap e644: 8 total, 8 up, 8 in 2024-03-30T14:11:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: osdmap e645: 8 total, 8 up, 8 in 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-126"}]: dispatch 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: pgmap v987: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: osdmap e644: 8 total, 8 up, 8 in 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: osdmap e645: 8 total, 8 up, 8 in 2024-03-30T14:11:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-126"}]: dispatch 2024-03-30T14:11:13.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:13 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:13.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-126"}]': finished 2024-03-30T14:11:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:13 smithi138 ceph-mon[33397]: osdmap e646: 8 total, 8 up, 8 in 2024-03-30T14:11:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-126", "mode": "writeback"}]: dispatch 2024-03-30T14:11:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-126"}]': finished 2024-03-30T14:11:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[37717]: osdmap e646: 8 total, 8 up, 8 in 2024-03-30T14:11:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-126", "mode": "writeback"}]: dispatch 2024-03-30T14:11:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-126"}]': finished 2024-03-30T14:11:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[29392]: osdmap e646: 8 total, 8 up, 8 in 2024-03-30T14:11:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-126", "mode": "writeback"}]: dispatch 2024-03-30T14:11:14.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:14 smithi138 ceph-mon[33397]: pgmap v990: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:14.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:14 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:14.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-126", "mode": "writeback"}]': finished 2024-03-30T14:11:14.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:14 smithi138 ceph-mon[33397]: osdmap e647: 8 total, 8 up, 8 in 2024-03-30T14:11:14.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[37717]: pgmap v990: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-126", "mode": "writeback"}]': finished 2024-03-30T14:11:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[37717]: osdmap e647: 8 total, 8 up, 8 in 2024-03-30T14:11:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:14.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[29392]: pgmap v990: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:14.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:14.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-126", "mode": "writeback"}]': finished 2024-03-30T14:11:14.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[29392]: osdmap e647: 8 total, 8 up, 8 in 2024-03-30T14:11:14.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:15.955 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:15 smithi138 ceph-mon[33397]: pgmap v993: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:15.955 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:15.956 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:15 smithi138 ceph-mon[33397]: osdmap e648: 8 total, 8 up, 8 in 2024-03-30T14:11:15.956 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126"}]: dispatch 2024-03-30T14:11:16.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[29392]: pgmap v993: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:16.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:16.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[29392]: osdmap e648: 8 total, 8 up, 8 in 2024-03-30T14:11:16.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126"}]: dispatch 2024-03-30T14:11:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[37717]: pgmap v993: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[37717]: osdmap e648: 8 total, 8 up, 8 in 2024-03-30T14:11:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126"}]: dispatch 2024-03-30T14:11:16.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:11:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:11:16.697 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:11:16.697 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9067 ms) 2024-03-30T14:11:16.697 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-30T14:11:16.697 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:11:16.697 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6046 ms) 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5042 ms) 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (24273 ms) 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14060 ms) 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12058 ms) 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-30T14:11:16.698 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: 00000012 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12079 ms) 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6056 ms) 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (556759 ms total) 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1024 ms) 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11060 ms) 2024-03-30T14:11:16.699 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18095 ms) 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (1 ms) 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30180 ms total) 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7055 ms) 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8044 ms) 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: waiting for scrub... 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: done waiting 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24291 ms) 2024-03-30T14:11:16.700 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9103 ms) 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8076 ms) 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8059 ms) 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10088 ms) 2024-03-30T14:11:16.701 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-30T14:11:16.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:11:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126"}]': finished 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[29392]: osdmap e649: 8 total, 8 up, 8 in 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126"}]': finished 2024-03-30T14:11:16.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[37717]: osdmap e649: 8 total, 8 up, 8 in 2024-03-30T14:11:16.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:16.936 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:16 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:16 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-126"}]': finished 2024-03-30T14:11:17.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:16 smithi138 ceph-mon[33397]: osdmap e649: 8 total, 8 up, 8 in 2024-03-30T14:11:17.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:17.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:16 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:17 smithi138 ceph-mon[33397]: pgmap v996: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:11:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:17 smithi138 ceph-mon[33397]: osdmap e650: 8 total, 8 up, 8 in 2024-03-30T14:11:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:17 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:11:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:17 smithi102 ceph-mon[37717]: pgmap v996: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:11:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:17 smithi102 ceph-mon[37717]: osdmap e650: 8 total, 8 up, 8 in 2024-03-30T14:11:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:17 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:11:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:17 smithi102 ceph-mon[29392]: pgmap v996: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:11:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:17 smithi102 ceph-mon[29392]: osdmap e650: 8 total, 8 up, 8 in 2024-03-30T14:11:18.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:17 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:11:19.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:18 smithi138 ceph-mon[33397]: osdmap e651: 8 total, 8 up, 8 in 2024-03-30T14:11:19.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:19.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:18 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:11:19.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:18 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:11:19.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:18 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:19.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[29392]: osdmap e651: 8 total, 8 up, 8 in 2024-03-30T14:11:19.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:19.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:11:19.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:11:19.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[37717]: osdmap e651: 8 total, 8 up, 8 in 2024-03-30T14:11:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:11:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:11:19.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:18 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:19 smithi138 ceph-mon[33397]: pgmap v999: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:19 smithi138 ceph-mon[33397]: osdmap e652: 8 total, 8 up, 8 in 2024-03-30T14:11:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[29392]: pgmap v999: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[29392]: osdmap e652: 8 total, 8 up, 8 in 2024-03-30T14:11:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:20.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[37717]: pgmap v999: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[37717]: osdmap e652: 8 total, 8 up, 8 in 2024-03-30T14:11:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:21.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:20 smithi138 ceph-mon[33397]: osdmap e653: 8 total, 8 up, 8 in 2024-03-30T14:11:21.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-128"}]: dispatch 2024-03-30T14:11:21.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-128"}]': finished 2024-03-30T14:11:21.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:20 smithi138 ceph-mon[33397]: osdmap e654: 8 total, 8 up, 8 in 2024-03-30T14:11:21.082 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-128", "mode": "writeback"}]: dispatch 2024-03-30T14:11:21.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[37717]: osdmap e653: 8 total, 8 up, 8 in 2024-03-30T14:11:21.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-128"}]: dispatch 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-128"}]': finished 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[37717]: osdmap e654: 8 total, 8 up, 8 in 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-128", "mode": "writeback"}]: dispatch 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[29392]: osdmap e653: 8 total, 8 up, 8 in 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-128"}]: dispatch 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-128"}]': finished 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[29392]: osdmap e654: 8 total, 8 up, 8 in 2024-03-30T14:11:21.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-128", "mode": "writeback"}]: dispatch 2024-03-30T14:11:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:21 smithi138 ceph-mon[33397]: pgmap v1002: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:11:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:21 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:22.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-128", "mode": "writeback"}]': finished 2024-03-30T14:11:22.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:21 smithi138 ceph-mon[33397]: osdmap e655: 8 total, 8 up, 8 in 2024-03-30T14:11:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[29392]: pgmap v1002: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:11:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-128", "mode": "writeback"}]': finished 2024-03-30T14:11:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[29392]: osdmap e655: 8 total, 8 up, 8 in 2024-03-30T14:11:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[37717]: pgmap v1002: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T14:11:22.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:22.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-128", "mode": "writeback"}]': finished 2024-03-30T14:11:22.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:21 smithi102 ceph-mon[37717]: osdmap e655: 8 total, 8 up, 8 in 2024-03-30T14:11:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:23 smithi138 ceph-mon[33397]: pgmap v1005: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:24.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:23 smithi102 ceph-mon[37717]: pgmap v1005: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:24.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:23 smithi102 ceph-mon[29392]: pgmap v1005: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:26.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:25 smithi138 ceph-mon[33397]: pgmap v1006: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-30T14:11:26.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:11:25 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:11:26.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:25 smithi102 ceph-mon[29392]: pgmap v1006: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-30T14:11:26.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:25 smithi102 ceph-mon[37717]: pgmap v1006: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-30T14:11:26.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:26.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:26.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:11:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:11:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:11:27.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:28.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[29392]: pgmap v1007: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 302 B/s wr, 1 op/s 2024-03-30T14:11:28.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:28.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:28.143 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:28.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[37717]: pgmap v1007: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 302 B/s wr, 1 op/s 2024-03-30T14:11:28.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:28.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:28.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:27 smithi138 ceph-mon[33397]: pgmap v1007: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 302 B/s wr, 1 op/s 2024-03-30T14:11:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:28.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:29.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:29.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:28 smithi138 ceph-mon[33397]: osdmap e656: 8 total, 8 up, 8 in 2024-03-30T14:11:29.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128"}]: dispatch 2024-03-30T14:11:29.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:29.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:28 smithi102 ceph-mon[29392]: osdmap e656: 8 total, 8 up, 8 in 2024-03-30T14:11:29.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128"}]: dispatch 2024-03-30T14:11:29.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:29.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:28 smithi102 ceph-mon[37717]: osdmap e656: 8 total, 8 up, 8 in 2024-03-30T14:11:29.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128"}]: dispatch 2024-03-30T14:11:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:30 smithi138 ceph-mon[33397]: pgmap v1009: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 789 B/s rd, 263 B/s wr, 1 op/s 2024-03-30T14:11:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:30 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128"}]': finished 2024-03-30T14:11:30.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:30 smithi138 ceph-mon[33397]: osdmap e657: 8 total, 8 up, 8 in 2024-03-30T14:11:30.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[29392]: pgmap v1009: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 789 B/s rd, 263 B/s wr, 1 op/s 2024-03-30T14:11:30.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:30.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128"}]': finished 2024-03-30T14:11:30.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[29392]: osdmap e657: 8 total, 8 up, 8 in 2024-03-30T14:11:30.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[37717]: pgmap v1009: 236 pgs: 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 789 B/s rd, 263 B/s wr, 1 op/s 2024-03-30T14:11:30.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:30.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-128"}]': finished 2024-03-30T14:11:30.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:30 smithi102 ceph-mon[37717]: osdmap e657: 8 total, 8 up, 8 in 2024-03-30T14:11:31.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:31 smithi138 ceph-mon[33397]: osdmap e658: 8 total, 8 up, 8 in 2024-03-30T14:11:31.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:31 smithi138 ceph-mon[33397]: osdmap e659: 8 total, 8 up, 8 in 2024-03-30T14:11:31.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:31.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:31 smithi102 ceph-mon[37717]: osdmap e658: 8 total, 8 up, 8 in 2024-03-30T14:11:31.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:31 smithi102 ceph-mon[37717]: osdmap e659: 8 total, 8 up, 8 in 2024-03-30T14:11:31.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:31 smithi102 ceph-mon[29392]: osdmap e658: 8 total, 8 up, 8 in 2024-03-30T14:11:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:31 smithi102 ceph-mon[29392]: osdmap e659: 8 total, 8 up, 8 in 2024-03-30T14:11:31.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:32.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:32 smithi138 ceph-mon[33397]: pgmap v1012: 204 pgs: 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:11:32.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:32.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:32 smithi138 ceph-mon[33397]: osdmap e660: 8 total, 8 up, 8 in 2024-03-30T14:11:32.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:32.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[29392]: pgmap v1012: 204 pgs: 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:11:32.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:32.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[29392]: osdmap e660: 8 total, 8 up, 8 in 2024-03-30T14:11:32.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:32.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[37717]: pgmap v1012: 204 pgs: 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:11:32.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:32.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[37717]: osdmap e660: 8 total, 8 up, 8 in 2024-03-30T14:11:32.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:34.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:33 smithi138 ceph-mon[33397]: pgmap v1015: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:11:34.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:34.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:33 smithi138 ceph-mon[33397]: osdmap e661: 8 total, 8 up, 8 in 2024-03-30T14:11:34.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-130"}]: dispatch 2024-03-30T14:11:34.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[29392]: pgmap v1015: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:11:34.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:34.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[29392]: osdmap e661: 8 total, 8 up, 8 in 2024-03-30T14:11:34.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-130"}]: dispatch 2024-03-30T14:11:34.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[37717]: pgmap v1015: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:11:34.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:34.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[37717]: osdmap e661: 8 total, 8 up, 8 in 2024-03-30T14:11:34.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-130"}]: dispatch 2024-03-30T14:11:35.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-130"}]': finished 2024-03-30T14:11:35.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:34 smithi138 ceph-mon[33397]: osdmap e662: 8 total, 8 up, 8 in 2024-03-30T14:11:35.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-130", "mode": "writeback"}]: dispatch 2024-03-30T14:11:35.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:34 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:35.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-130", "mode": "writeback"}]': finished 2024-03-30T14:11:35.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:34 smithi138 ceph-mon[33397]: osdmap e663: 8 total, 8 up, 8 in 2024-03-30T14:11:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-130"}]': finished 2024-03-30T14:11:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[37717]: osdmap e662: 8 total, 8 up, 8 in 2024-03-30T14:11:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-130", "mode": "writeback"}]: dispatch 2024-03-30T14:11:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-130", "mode": "writeback"}]': finished 2024-03-30T14:11:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[37717]: osdmap e663: 8 total, 8 up, 8 in 2024-03-30T14:11:35.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-130"}]': finished 2024-03-30T14:11:35.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[29392]: osdmap e662: 8 total, 8 up, 8 in 2024-03-30T14:11:35.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-130", "mode": "writeback"}]: dispatch 2024-03-30T14:11:35.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:35.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-130", "mode": "writeback"}]': finished 2024-03-30T14:11:35.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:34 smithi102 ceph-mon[29392]: osdmap e663: 8 total, 8 up, 8 in 2024-03-30T14:11:36.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:11:35 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:11:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:35 smithi138 ceph-mon[33397]: pgmap v1018: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:35 smithi102 ceph-mon[29392]: pgmap v1018: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:35 smithi102 ceph-mon[37717]: pgmap v1018: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:36.789 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:11:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:11:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:37.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:36 smithi138 ceph-mon[33397]: osdmap e664: 8 total, 8 up, 8 in 2024-03-30T14:11:37.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130"}]: dispatch 2024-03-30T14:11:37.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[29392]: osdmap e664: 8 total, 8 up, 8 in 2024-03-30T14:11:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130"}]: dispatch 2024-03-30T14:11:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:37.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:37.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[37717]: osdmap e664: 8 total, 8 up, 8 in 2024-03-30T14:11:37.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130"}]: dispatch 2024-03-30T14:11:37.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:38.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:38 smithi138 ceph-mon[33397]: pgmap v1021: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T14:11:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:38 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:38 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130"}]': finished 2024-03-30T14:11:38.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:38 smithi138 ceph-mon[33397]: osdmap e665: 8 total, 8 up, 8 in 2024-03-30T14:11:38.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[29392]: pgmap v1021: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T14:11:38.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:38.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130"}]': finished 2024-03-30T14:11:38.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[29392]: osdmap e665: 8 total, 8 up, 8 in 2024-03-30T14:11:38.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[37717]: pgmap v1021: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T14:11:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-130"}]': finished 2024-03-30T14:11:38.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:38 smithi102 ceph-mon[37717]: osdmap e665: 8 total, 8 up, 8 in 2024-03-30T14:11:39.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:39 smithi138 ceph-mon[33397]: osdmap e666: 8 total, 8 up, 8 in 2024-03-30T14:11:39.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:39 smithi138 ceph-mon[33397]: osdmap e667: 8 total, 8 up, 8 in 2024-03-30T14:11:39.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:39.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:39 smithi102 ceph-mon[29392]: osdmap e666: 8 total, 8 up, 8 in 2024-03-30T14:11:39.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:39 smithi102 ceph-mon[29392]: osdmap e667: 8 total, 8 up, 8 in 2024-03-30T14:11:39.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:39.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:39 smithi102 ceph-mon[37717]: osdmap e666: 8 total, 8 up, 8 in 2024-03-30T14:11:39.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:39 smithi102 ceph-mon[37717]: osdmap e667: 8 total, 8 up, 8 in 2024-03-30T14:11:39.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:40.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:40 smithi138 ceph-mon[33397]: pgmap v1024: 204 pgs: 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:40 smithi138 ceph-mon[33397]: osdmap e668: 8 total, 8 up, 8 in 2024-03-30T14:11:40.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:40.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[29392]: pgmap v1024: 204 pgs: 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:40.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:40.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[29392]: osdmap e668: 8 total, 8 up, 8 in 2024-03-30T14:11:40.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[37717]: pgmap v1024: 204 pgs: 204 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:11:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[37717]: osdmap e668: 8 total, 8 up, 8 in 2024-03-30T14:11:40.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:41 smithi138 ceph-mon[33397]: pgmap v1027: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:11:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:41 smithi138 ceph-mon[33397]: osdmap e669: 8 total, 8 up, 8 in 2024-03-30T14:11:42.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[29392]: pgmap v1027: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:11:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[29392]: osdmap e669: 8 total, 8 up, 8 in 2024-03-30T14:11:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:42.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[37717]: pgmap v1027: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:11:42.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:42.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[37717]: osdmap e669: 8 total, 8 up, 8 in 2024-03-30T14:11:42.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:43.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:43.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:42 smithi138 ceph-mon[33397]: osdmap e670: 8 total, 8 up, 8 in 2024-03-30T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-132", "mode": "writeback"}]: dispatch 2024-03-30T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:42 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:42 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:42 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-132", "mode": "writeback"}]': finished 2024-03-30T14:11:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[37717]: osdmap e670: 8 total, 8 up, 8 in 2024-03-30T14:11:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-132", "mode": "writeback"}]: dispatch 2024-03-30T14:11:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-132", "mode": "writeback"}]': finished 2024-03-30T14:11:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[29392]: osdmap e670: 8 total, 8 up, 8 in 2024-03-30T14:11:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-132", "mode": "writeback"}]: dispatch 2024-03-30T14:11:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:43.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:42 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-132", "mode": "writeback"}]': finished 2024-03-30T14:11:44.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:43 smithi138 ceph-mon[33397]: pgmap v1030: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:44.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:43 smithi138 ceph-mon[33397]: osdmap e671: 8 total, 8 up, 8 in 2024-03-30T14:11:44.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:43 smithi102 ceph-mon[37717]: pgmap v1030: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:44.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:43 smithi102 ceph-mon[37717]: osdmap e671: 8 total, 8 up, 8 in 2024-03-30T14:11:44.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:43 smithi102 ceph-mon[29392]: pgmap v1030: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:44.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:43 smithi102 ceph-mon[29392]: osdmap e671: 8 total, 8 up, 8 in 2024-03-30T14:11:45.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:44 smithi138 ceph-mon[33397]: osdmap e672: 8 total, 8 up, 8 in 2024-03-30T14:11:45.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:44 smithi102 ceph-mon[29392]: osdmap e672: 8 total, 8 up, 8 in 2024-03-30T14:11:45.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:44 smithi102 ceph-mon[37717]: osdmap e672: 8 total, 8 up, 8 in 2024-03-30T14:11:46.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:11:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:11:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:45 smithi138 ceph-mon[33397]: pgmap v1033: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:45 smithi138 ceph-mon[33397]: osdmap e673: 8 total, 8 up, 8 in 2024-03-30T14:11:46.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:46.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:45 smithi102 ceph-mon[29392]: pgmap v1033: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:46.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:45 smithi102 ceph-mon[29392]: osdmap e673: 8 total, 8 up, 8 in 2024-03-30T14:11:46.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:46.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:45 smithi102 ceph-mon[37717]: pgmap v1033: 236 pgs: 236 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:46.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:45 smithi102 ceph-mon[37717]: osdmap e673: 8 total, 8 up, 8 in 2024-03-30T14:11:46.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:46.842 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:11:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:11:47.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:47.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:46 smithi138 ceph-mon[33397]: osdmap e674: 8 total, 8 up, 8 in 2024-03-30T14:11:47.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:47.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:47.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:47.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[29392]: osdmap e674: 8 total, 8 up, 8 in 2024-03-30T14:11:47.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:47.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:47.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:47.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[37717]: osdmap e674: 8 total, 8 up, 8 in 2024-03-30T14:11:47.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:47.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:47 smithi138 ceph-mon[33397]: pgmap v1036: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:11:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:47 smithi138 ceph-mon[33397]: osdmap e675: 8 total, 8 up, 8 in 2024-03-30T14:11:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:47 smithi102 ceph-mon[29392]: pgmap v1036: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:11:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:47 smithi102 ceph-mon[29392]: osdmap e675: 8 total, 8 up, 8 in 2024-03-30T14:11:48.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:47 smithi102 ceph-mon[37717]: pgmap v1036: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:11:48.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:48.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:47 smithi102 ceph-mon[37717]: osdmap e675: 8 total, 8 up, 8 in 2024-03-30T14:11:49.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:48 smithi138 ceph-mon[33397]: osdmap e676: 8 total, 8 up, 8 in 2024-03-30T14:11:49.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:49.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:48 smithi102 ceph-mon[29392]: osdmap e676: 8 total, 8 up, 8 in 2024-03-30T14:11:49.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:49.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:48 smithi102 ceph-mon[37717]: osdmap e676: 8 total, 8 up, 8 in 2024-03-30T14:11:49.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:50.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:49 smithi138 ceph-mon[33397]: pgmap v1039: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:11:50.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:50.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:49 smithi138 ceph-mon[33397]: osdmap e677: 8 total, 8 up, 8 in 2024-03-30T14:11:50.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:50.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[29392]: pgmap v1039: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:11:50.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:50.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[29392]: osdmap e677: 8 total, 8 up, 8 in 2024-03-30T14:11:50.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[37717]: pgmap v1039: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-30T14:11:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[37717]: osdmap e677: 8 total, 8 up, 8 in 2024-03-30T14:11:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132"}]: dispatch 2024-03-30T14:11:51.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:50 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:51.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:51.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:50 smithi138 ceph-mon[33397]: osdmap e678: 8 total, 8 up, 8 in 2024-03-30T14:11:51.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:50 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:51.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:51.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:50 smithi102 ceph-mon[29392]: osdmap e678: 8 total, 8 up, 8 in 2024-03-30T14:11:51.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:50 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:51.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-132"}]': finished 2024-03-30T14:11:51.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:50 smithi102 ceph-mon[37717]: osdmap e678: 8 total, 8 up, 8 in 2024-03-30T14:11:52.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:51 smithi138 ceph-mon[33397]: pgmap v1042: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:11:52.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:51 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:51 smithi138 ceph-mon[33397]: osdmap e679: 8 total, 8 up, 8 in 2024-03-30T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:51 smithi138 ceph-mon[33397]: osdmap e680: 8 total, 8 up, 8 in 2024-03-30T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:51 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:52.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[29392]: pgmap v1042: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:11:52.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:52.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[29392]: osdmap e679: 8 total, 8 up, 8 in 2024-03-30T14:11:52.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[29392]: osdmap e680: 8 total, 8 up, 8 in 2024-03-30T14:11:52.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[37717]: pgmap v1042: 236 pgs: 236 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:11:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[37717]: osdmap e679: 8 total, 8 up, 8 in 2024-03-30T14:11:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[37717]: osdmap e680: 8 total, 8 up, 8 in 2024-03-30T14:11:52.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:51 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:11:54.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:53 smithi138 ceph-mon[33397]: pgmap v1045: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:53 smithi138 ceph-mon[33397]: osdmap e681: 8 total, 8 up, 8 in 2024-03-30T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[29392]: pgmap v1045: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[29392]: osdmap e681: 8 total, 8 up, 8 in 2024-03-30T14:11:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[37717]: pgmap v1045: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:11:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[37717]: osdmap e681: 8 total, 8 up, 8 in 2024-03-30T14:11:54.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:11:55.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:55.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:54 smithi138 ceph-mon[33397]: osdmap e682: 8 total, 8 up, 8 in 2024-03-30T14:11:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-134"}]: dispatch 2024-03-30T14:11:55.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-134"}]': finished 2024-03-30T14:11:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[29392]: osdmap e682: 8 total, 8 up, 8 in 2024-03-30T14:11:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-134"}]: dispatch 2024-03-30T14:11:55.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-134"}]': finished 2024-03-30T14:11:55.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:11:55.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[37717]: osdmap e682: 8 total, 8 up, 8 in 2024-03-30T14:11:55.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-134"}]: dispatch 2024-03-30T14:11:55.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-134"}]': finished 2024-03-30T14:11:55.993 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:55 smithi138 ceph-mon[33397]: pgmap v1048: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:55.994 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:55 smithi138 ceph-mon[33397]: osdmap e683: 8 total, 8 up, 8 in 2024-03-30T14:11:55.994 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-134", "mode": "writeback"}]: dispatch 2024-03-30T14:11:55.994 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:55 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:55.994 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-134", "mode": "writeback"}]': finished 2024-03-30T14:11:55.994 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:55 smithi138 ceph-mon[33397]: osdmap e684: 8 total, 8 up, 8 in 2024-03-30T14:11:56.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[29392]: pgmap v1048: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[29392]: osdmap e683: 8 total, 8 up, 8 in 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-134", "mode": "writeback"}]: dispatch 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-134", "mode": "writeback"}]': finished 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[29392]: osdmap e684: 8 total, 8 up, 8 in 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[37717]: pgmap v1048: 236 pgs: 32 unknown, 204 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[37717]: osdmap e683: 8 total, 8 up, 8 in 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-134", "mode": "writeback"}]: dispatch 2024-03-30T14:11:56.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:11:56.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-134", "mode": "writeback"}]': finished 2024-03-30T14:11:56.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:55 smithi102 ceph-mon[37717]: osdmap e684: 8 total, 8 up, 8 in 2024-03-30T14:11:56.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:11:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:11:56.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:11:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:11:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:11:56.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:56.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:56.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:56 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:56.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:56.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:56.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:56 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:57.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:11:57.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:11:57.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:56 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:11:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:57 smithi138 ceph-mon[33397]: pgmap v1051: 236 pgs: 236 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:57 smithi138 ceph-mon[33397]: osdmap e685: 8 total, 8 up, 8 in 2024-03-30T14:11:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:57 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134"}]: dispatch 2024-03-30T14:11:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:57 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[29392]: pgmap v1051: 236 pgs: 236 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[29392]: osdmap e685: 8 total, 8 up, 8 in 2024-03-30T14:11:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134"}]: dispatch 2024-03-30T14:11:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[37717]: pgmap v1051: 236 pgs: 236 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:11:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:11:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[37717]: osdmap e685: 8 total, 8 up, 8 in 2024-03-30T14:11:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134"}]: dispatch 2024-03-30T14:11:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:11:58.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:11:59.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:58 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:58 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134"}]': finished 2024-03-30T14:11:59.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:58 smithi138 ceph-mon[33397]: osdmap e686: 8 total, 8 up, 8 in 2024-03-30T14:11:59.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:58 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:59.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:58 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134"}]': finished 2024-03-30T14:11:59.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:58 smithi102 ceph-mon[29392]: osdmap e686: 8 total, 8 up, 8 in 2024-03-30T14:11:59.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:58 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:11:59.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:58 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-134"}]': finished 2024-03-30T14:11:59.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:58 smithi102 ceph-mon[37717]: osdmap e686: 8 total, 8 up, 8 in 2024-03-30T14:12:00.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:59 smithi138 ceph-mon[33397]: pgmap v1054: 236 pgs: 236 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:00.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:11:59 smithi138 ceph-mon[33397]: osdmap e687: 8 total, 8 up, 8 in 2024-03-30T14:12:00.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:59 smithi102 ceph-mon[29392]: pgmap v1054: 236 pgs: 236 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:00.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:11:59 smithi102 ceph-mon[29392]: osdmap e687: 8 total, 8 up, 8 in 2024-03-30T14:12:00.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:59 smithi102 ceph-mon[37717]: pgmap v1054: 236 pgs: 236 active+clean; 582 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:00.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:11:59 smithi102 ceph-mon[37717]: osdmap e687: 8 total, 8 up, 8 in 2024-03-30T14:12:01.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:00 smithi138 ceph-mon[33397]: osdmap e688: 8 total, 8 up, 8 in 2024-03-30T14:12:01.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:00 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:01.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:00 smithi138 ceph-mon[33397]: osdmap e689: 8 total, 8 up, 8 in 2024-03-30T14:12:01.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[29392]: osdmap e688: 8 total, 8 up, 8 in 2024-03-30T14:12:01.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:01.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:01.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[29392]: osdmap e689: 8 total, 8 up, 8 in 2024-03-30T14:12:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[37717]: osdmap e688: 8 total, 8 up, 8 in 2024-03-30T14:12:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:01.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:00 smithi102 ceph-mon[37717]: osdmap e689: 8 total, 8 up, 8 in 2024-03-30T14:12:02.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:01 smithi138 ceph-mon[33397]: pgmap v1057: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:01 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:01 smithi138 ceph-mon[33397]: osdmap e690: 8 total, 8 up, 8 in 2024-03-30T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:01 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-136"}]: dispatch 2024-03-30T14:12:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[29392]: pgmap v1057: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:12:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[29392]: osdmap e690: 8 total, 8 up, 8 in 2024-03-30T14:12:02.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-136"}]: dispatch 2024-03-30T14:12:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[37717]: pgmap v1057: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:12:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[37717]: osdmap e690: 8 total, 8 up, 8 in 2024-03-30T14:12:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:01 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-136"}]: dispatch 2024-03-30T14:12:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:03 smithi138 ceph-mon[33397]: pgmap v1060: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:12:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-136"}]': finished 2024-03-30T14:12:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:03 smithi138 ceph-mon[33397]: osdmap e691: 8 total, 8 up, 8 in 2024-03-30T14:12:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:03 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-136", "mode": "writeback"}]: dispatch 2024-03-30T14:12:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[29392]: pgmap v1060: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:12:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-136"}]': finished 2024-03-30T14:12:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[29392]: osdmap e691: 8 total, 8 up, 8 in 2024-03-30T14:12:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-136", "mode": "writeback"}]: dispatch 2024-03-30T14:12:04.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[37717]: pgmap v1060: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T14:12:04.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-136"}]': finished 2024-03-30T14:12:04.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[37717]: osdmap e691: 8 total, 8 up, 8 in 2024-03-30T14:12:04.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:03 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-136", "mode": "writeback"}]: dispatch 2024-03-30T14:12:05.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:04 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:05.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-136", "mode": "writeback"}]': finished 2024-03-30T14:12:05.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:04 smithi138 ceph-mon[33397]: osdmap e692: 8 total, 8 up, 8 in 2024-03-30T14:12:05.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:04 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-136", "mode": "writeback"}]': finished 2024-03-30T14:12:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[29392]: osdmap e692: 8 total, 8 up, 8 in 2024-03-30T14:12:05.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:05.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:05.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-136", "mode": "writeback"}]': finished 2024-03-30T14:12:05.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[37717]: osdmap e692: 8 total, 8 up, 8 in 2024-03-30T14:12:05.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:04 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:06.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:12:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:12:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:05 smithi138 ceph-mon[33397]: pgmap v1063: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:12:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:05 smithi138 ceph-mon[33397]: osdmap e693: 8 total, 8 up, 8 in 2024-03-30T14:12:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:05 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136"}]: dispatch 2024-03-30T14:12:06.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[37717]: pgmap v1063: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:06.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:12:06.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[37717]: osdmap e693: 8 total, 8 up, 8 in 2024-03-30T14:12:06.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136"}]: dispatch 2024-03-30T14:12:06.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[29392]: pgmap v1063: 236 pgs: 32 creating+peering, 204 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:06.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:12:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[29392]: osdmap e693: 8 total, 8 up, 8 in 2024-03-30T14:12:06.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:05 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136"}]: dispatch 2024-03-30T14:12:06.776 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:12:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:12:07.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:06 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:12:07.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136"}]': finished 2024-03-30T14:12:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:06 smithi138 ceph-mon[33397]: osdmap e694: 8 total, 8 up, 8 in 2024-03-30T14:12:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:07.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:06 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:12:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136"}]': finished 2024-03-30T14:12:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[37717]: osdmap e694: 8 total, 8 up, 8 in 2024-03-30T14:12:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:07.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:12:07.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-136"}]': finished 2024-03-30T14:12:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[29392]: osdmap e694: 8 total, 8 up, 8 in 2024-03-30T14:12:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:07.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:06 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:07 smithi138 ceph-mon[33397]: pgmap v1066: 236 pgs: 236 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-30T14:12:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:07 smithi138 ceph-mon[33397]: osdmap e695: 8 total, 8 up, 8 in 2024-03-30T14:12:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:07 smithi102 ceph-mon[37717]: pgmap v1066: 236 pgs: 236 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-30T14:12:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:07 smithi102 ceph-mon[37717]: osdmap e695: 8 total, 8 up, 8 in 2024-03-30T14:12:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:07 smithi102 ceph-mon[29392]: pgmap v1066: 236 pgs: 236 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-30T14:12:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:07 smithi102 ceph-mon[29392]: osdmap e695: 8 total, 8 up, 8 in 2024-03-30T14:12:09.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:08 smithi138 ceph-mon[33397]: osdmap e696: 8 total, 8 up, 8 in 2024-03-30T14:12:09.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:08 smithi102 ceph-mon[37717]: osdmap e696: 8 total, 8 up, 8 in 2024-03-30T14:12:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:08 smithi102 ceph-mon[29392]: osdmap e696: 8 total, 8 up, 8 in 2024-03-30T14:12:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:09 smithi138 ceph-mon[33397]: pgmap v1069: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-30T14:12:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:09 smithi138 ceph-mon[33397]: osdmap e697: 8 total, 8 up, 8 in 2024-03-30T14:12:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[37717]: pgmap v1069: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[37717]: osdmap e697: 8 total, 8 up, 8 in 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[29392]: pgmap v1069: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[29392]: osdmap e697: 8 total, 8 up, 8 in 2024-03-30T14:12:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:11.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:11.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:10 smithi138 ceph-mon[33397]: osdmap e698: 8 total, 8 up, 8 in 2024-03-30T14:12:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:12:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:12:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:10 smithi138 ceph-mon[33397]: osdmap e699: 8 total, 8 up, 8 in 2024-03-30T14:12:11.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:12:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[37717]: osdmap e698: 8 total, 8 up, 8 in 2024-03-30T14:12:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:12:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:12:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[37717]: osdmap e699: 8 total, 8 up, 8 in 2024-03-30T14:12:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:12:11.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[29392]: osdmap e698: 8 total, 8 up, 8 in 2024-03-30T14:12:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:12:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:12:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[29392]: osdmap e699: 8 total, 8 up, 8 in 2024-03-30T14:12:11.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:12:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:11 smithi138 ceph-mon[33397]: pgmap v1072: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:12:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:11 smithi138 ceph-mon[33397]: osdmap e700: 8 total, 8 up, 8 in 2024-03-30T14:12:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:12:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[37717]: pgmap v1072: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:12:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[37717]: osdmap e700: 8 total, 8 up, 8 in 2024-03-30T14:12:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:12:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[29392]: pgmap v1072: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:12:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[29392]: osdmap e700: 8 total, 8 up, 8 in 2024-03-30T14:12:12.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T14:12:13.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:13 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:13 smithi138 ceph-mon[33397]: pgmap v1075: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:12:13.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:13 smithi138 ceph-mon[33397]: osdmap e701: 8 total, 8 up, 8 in 2024-03-30T14:12:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[37717]: pgmap v1075: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:12:13.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[37717]: osdmap e701: 8 total, 8 up, 8 in 2024-03-30T14:12:13.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[29392]: pgmap v1075: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T14:12:13.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:13 smithi102 ceph-mon[29392]: osdmap e701: 8 total, 8 up, 8 in 2024-03-30T14:12:14.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:14.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138"}]: dispatch 2024-03-30T14:12:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138"}]: dispatch 2024-03-30T14:12:14.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:14.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138"}]: dispatch 2024-03-30T14:12:15.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138"}]': finished 2024-03-30T14:12:15.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:15 smithi138 ceph-mon[33397]: osdmap e702: 8 total, 8 up, 8 in 2024-03-30T14:12:15.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:15 smithi138 ceph-mon[33397]: pgmap v1078: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:15.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138"}]': finished 2024-03-30T14:12:15.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:15 smithi102 ceph-mon[37717]: osdmap e702: 8 total, 8 up, 8 in 2024-03-30T14:12:15.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:15 smithi102 ceph-mon[37717]: pgmap v1078: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:15.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-138"}]': finished 2024-03-30T14:12:15.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:15 smithi102 ceph-mon[29392]: osdmap e702: 8 total, 8 up, 8 in 2024-03-30T14:12:15.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:15 smithi102 ceph-mon[29392]: pgmap v1078: 236 pgs: 236 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:16.354 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:12:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:12:16.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:16 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:16.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:16 smithi138 ceph-mon[33397]: osdmap e703: 8 total, 8 up, 8 in 2024-03-30T14:12:16.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:16.684 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:12:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:12:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:12:16.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:16 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:16.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:16 smithi102 ceph-mon[37717]: osdmap e703: 8 total, 8 up, 8 in 2024-03-30T14:12:16.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:16.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:16 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:16.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:16 smithi102 ceph-mon[29392]: osdmap e703: 8 total, 8 up, 8 in 2024-03-30T14:12:16.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:17.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:17 smithi138 ceph-mon[33397]: osdmap e704: 8 total, 8 up, 8 in 2024-03-30T14:12:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:17.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:17 smithi138 ceph-mon[33397]: pgmap v1081: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:17.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:17 smithi102 ceph-mon[29392]: osdmap e704: 8 total, 8 up, 8 in 2024-03-30T14:12:17.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:17.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:17 smithi102 ceph-mon[29392]: pgmap v1081: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:17.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:17 smithi102 ceph-mon[37717]: osdmap e704: 8 total, 8 up, 8 in 2024-03-30T14:12:17.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:17.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:17 smithi102 ceph-mon[37717]: pgmap v1081: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:18.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:18.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:18 smithi138 ceph-mon[33397]: osdmap e705: 8 total, 8 up, 8 in 2024-03-30T14:12:18.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:18 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:18.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:18 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[29392]: osdmap e705: 8 total, 8 up, 8 in 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[37717]: osdmap e705: 8 total, 8 up, 8 in 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:18.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:18 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:12:19.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:19.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: osdmap e706: 8 total, 8 up, 8 in 2024-03-30T14:12:19.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:12:19.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: pgmap v1084: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:19.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:12:19.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:12:19.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:19 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:19.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: osdmap e706: 8 total, 8 up, 8 in 2024-03-30T14:12:19.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:12:19.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: pgmap v1084: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:19.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: osdmap e706: 8 total, 8 up, 8 in 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: pgmap v1084: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:12:19.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:19 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:20.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_count","val": "3"}]': finished 2024-03-30T14:12:20.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:20 smithi138 ceph-mon[33397]: osdmap e707: 8 total, 8 up, 8 in 2024-03-30T14:12:20.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:20 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:12:20.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_count","val": "3"}]': finished 2024-03-30T14:12:20.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:20 smithi102 ceph-mon[29392]: osdmap e707: 8 total, 8 up, 8 in 2024-03-30T14:12:20.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:20 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:12:20.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_count","val": "3"}]': finished 2024-03-30T14:12:20.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:20 smithi102 ceph-mon[37717]: osdmap e707: 8 total, 8 up, 8 in 2024-03-30T14:12:20.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:20 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T14:12:21.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_period","val": "3"}]': finished 2024-03-30T14:12:21.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:21 smithi138 ceph-mon[33397]: osdmap e708: 8 total, 8 up, 8 in 2024-03-30T14:12:21.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:21 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:12:21.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:21 smithi138 ceph-mon[33397]: pgmap v1087: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:21.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_period","val": "3"}]': finished 2024-03-30T14:12:21.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[37717]: osdmap e708: 8 total, 8 up, 8 in 2024-03-30T14:12:21.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:12:21.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[37717]: pgmap v1087: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:21.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_period","val": "3"}]': finished 2024-03-30T14:12:21.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[29392]: osdmap e708: 8 total, 8 up, 8 in 2024-03-30T14:12:21.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:12:21.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:21 smithi102 ceph-mon[29392]: pgmap v1087: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:22.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:12:22.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:22 smithi138 ceph-mon[33397]: osdmap e709: 8 total, 8 up, 8 in 2024-03-30T14:12:22.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:12:22.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:22 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:22.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:22 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T14:12:22.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:22 smithi138 ceph-mon[33397]: osdmap e710: 8 total, 8 up, 8 in 2024-03-30T14:12:22.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:12:22.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[37717]: osdmap e709: 8 total, 8 up, 8 in 2024-03-30T14:12:22.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:12:22.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T14:12:22.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[37717]: osdmap e710: 8 total, 8 up, 8 in 2024-03-30T14:12:22.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:12:22.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[29392]: osdmap e709: 8 total, 8 up, 8 in 2024-03-30T14:12:22.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T14:12:22.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:22.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T14:12:22.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:22 smithi102 ceph-mon[29392]: osdmap e710: 8 total, 8 up, 8 in 2024-03-30T14:12:23.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:23 smithi138 ceph-mon[33397]: pgmap v1090: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:23.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:23 smithi102 ceph-mon[29392]: pgmap v1090: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:23.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:23 smithi102 ceph-mon[37717]: pgmap v1090: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:25.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:25 smithi138 ceph-mon[33397]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-30T14:12:25.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:25 smithi102 ceph-mon[29392]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-30T14:12:25.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:25 smithi102 ceph-mon[37717]: pgmap v1091: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-30T14:12:26.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:12:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:12:26.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:26.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:26.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:12:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:12:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:12:26.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:27.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:27 smithi138 ceph-mon[33397]: pgmap v1092: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-30T14:12:27.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:27.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:27 smithi102 ceph-mon[29392]: pgmap v1092: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-30T14:12:27.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:27.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:27 smithi102 ceph-mon[37717]: pgmap v1092: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-30T14:12:27.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:29 smithi138 ceph-mon[33397]: pgmap v1093: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-30T14:12:29.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:29 smithi102 ceph-mon[29392]: pgmap v1093: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-30T14:12:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:29 smithi102 ceph-mon[37717]: pgmap v1093: 236 pgs: 236 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-30T14:12:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:31 smithi138 ceph-mon[33397]: pgmap v1094: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-03-30T14:12:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:31 smithi102 ceph-mon[29392]: pgmap v1094: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-03-30T14:12:31.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:31 smithi102 ceph-mon[37717]: pgmap v1094: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-03-30T14:12:33.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:33 smithi138 ceph-mon[33397]: pgmap v1095: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 6.5 KiB/s wr, 1 op/s 2024-03-30T14:12:33.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:33 smithi102 ceph-mon[29392]: pgmap v1095: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 6.5 KiB/s wr, 1 op/s 2024-03-30T14:12:33.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:33 smithi102 ceph-mon[37717]: pgmap v1095: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 6.5 KiB/s wr, 1 op/s 2024-03-30T14:12:34.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:34.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140"}]: dispatch 2024-03-30T14:12:34.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:34.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140"}]: dispatch 2024-03-30T14:12:34.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:34.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140"}]: dispatch 2024-03-30T14:12:36.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:12:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:12:36.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:35 smithi138 ceph-mon[33397]: pgmap v1096: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-03-30T14:12:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140"}]': finished 2024-03-30T14:12:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:35 smithi138 ceph-mon[33397]: osdmap e711: 8 total, 8 up, 8 in 2024-03-30T14:12:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:35 smithi102 ceph-mon[29392]: pgmap v1096: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-03-30T14:12:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140"}]': finished 2024-03-30T14:12:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:35 smithi102 ceph-mon[29392]: osdmap e711: 8 total, 8 up, 8 in 2024-03-30T14:12:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:35 smithi102 ceph-mon[37717]: pgmap v1096: 236 pgs: 236 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-03-30T14:12:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-140"}]': finished 2024-03-30T14:12:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:35 smithi102 ceph-mon[37717]: osdmap e711: 8 total, 8 up, 8 in 2024-03-30T14:12:36.842 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:12:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:12:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:36 smithi138 ceph-mon[33397]: osdmap e712: 8 total, 8 up, 8 in 2024-03-30T14:12:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:36 smithi138 ceph-mon[33397]: osdmap e713: 8 total, 8 up, 8 in 2024-03-30T14:12:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[29392]: osdmap e712: 8 total, 8 up, 8 in 2024-03-30T14:12:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[29392]: osdmap e713: 8 total, 8 up, 8 in 2024-03-30T14:12:37.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:37.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[37717]: osdmap e712: 8 total, 8 up, 8 in 2024-03-30T14:12:37.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:37.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[37717]: osdmap e713: 8 total, 8 up, 8 in 2024-03-30T14:12:37.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:38.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:37 smithi138 ceph-mon[33397]: pgmap v1099: 204 pgs: 204 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:38.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:38.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:37 smithi138 ceph-mon[33397]: osdmap e714: 8 total, 8 up, 8 in 2024-03-30T14:12:38.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[29392]: pgmap v1099: 204 pgs: 204 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[29392]: osdmap e714: 8 total, 8 up, 8 in 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[37717]: pgmap v1099: 204 pgs: 204 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[37717]: osdmap e714: 8 total, 8 up, 8 in 2024-03-30T14:12:38.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:39 smithi138 ceph-mon[33397]: pgmap v1102: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:39 smithi138 ceph-mon[33397]: osdmap e715: 8 total, 8 up, 8 in 2024-03-30T14:12:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:39.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[29392]: pgmap v1102: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:39.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:39.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[29392]: osdmap e715: 8 total, 8 up, 8 in 2024-03-30T14:12:39.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:39.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[37717]: pgmap v1102: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:39.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:39.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[37717]: osdmap e715: 8 total, 8 up, 8 in 2024-03-30T14:12:39.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:40.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-142"}]': finished 2024-03-30T14:12:40.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:40 smithi138 ceph-mon[33397]: osdmap e716: 8 total, 8 up, 8 in 2024-03-30T14:12:40.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-142", "mode": "writeback"}]: dispatch 2024-03-30T14:12:40.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:40 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:40.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-142", "mode": "writeback"}]': finished 2024-03-30T14:12:40.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:40 smithi138 ceph-mon[33397]: osdmap e717: 8 total, 8 up, 8 in 2024-03-30T14:12:40.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:12:40.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-142"}]': finished 2024-03-30T14:12:40.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[29392]: osdmap e716: 8 total, 8 up, 8 in 2024-03-30T14:12:40.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-142", "mode": "writeback"}]: dispatch 2024-03-30T14:12:40.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:40.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-142", "mode": "writeback"}]': finished 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[29392]: osdmap e717: 8 total, 8 up, 8 in 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-142"}]': finished 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[37717]: osdmap e716: 8 total, 8 up, 8 in 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-142", "mode": "writeback"}]: dispatch 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-142", "mode": "writeback"}]': finished 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[37717]: osdmap e717: 8 total, 8 up, 8 in 2024-03-30T14:12:40.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:12:41.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:41 smithi138 ceph-mon[33397]: pgmap v1105: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:12:41.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:12:41.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:41 smithi138 ceph-mon[33397]: osdmap e718: 8 total, 8 up, 8 in 2024-03-30T14:12:41.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:12:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[29392]: pgmap v1105: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:12:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:12:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[29392]: osdmap e718: 8 total, 8 up, 8 in 2024-03-30T14:12:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:12:41.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[37717]: pgmap v1105: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:12:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:12:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[37717]: osdmap e718: 8 total, 8 up, 8 in 2024-03-30T14:12:41.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:12:43.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:43 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:43.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:43.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:43 smithi138 ceph-mon[33397]: pgmap v1108: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:12:43.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:12:43.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:43 smithi138 ceph-mon[33397]: osdmap e719: 8 total, 8 up, 8 in 2024-03-30T14:12:43.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:12:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[29392]: pgmap v1108: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:12:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:12:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[29392]: osdmap e719: 8 total, 8 up, 8 in 2024-03-30T14:12:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:12:43.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:12:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[37717]: pgmap v1108: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T14:12:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:12:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[37717]: osdmap e719: 8 total, 8 up, 8 in 2024-03-30T14:12:43.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:12:44.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:44 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:12:44.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:12:44.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:44 smithi138 ceph-mon[33397]: osdmap e720: 8 total, 8 up, 8 in 2024-03-30T14:12:44.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:44 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:12:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:12:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:12:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[29392]: osdmap e720: 8 total, 8 up, 8 in 2024-03-30T14:12:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:12:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:12:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:12:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[37717]: osdmap e720: 8 total, 8 up, 8 in 2024-03-30T14:12:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:44 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:12:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:45 smithi138 ceph-mon[33397]: pgmap v1111: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:12:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:45 smithi138 ceph-mon[33397]: osdmap e721: 8 total, 8 up, 8 in 2024-03-30T14:12:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:12:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[29392]: pgmap v1111: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:12:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[29392]: osdmap e721: 8 total, 8 up, 8 in 2024-03-30T14:12:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:12:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[37717]: pgmap v1111: 236 pgs: 236 active+clean; 4.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:12:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[37717]: osdmap e721: 8 total, 8 up, 8 in 2024-03-30T14:12:45.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T14:12:46.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:12:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:12:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T14:12:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:46 smithi138 ceph-mon[33397]: osdmap e722: 8 total, 8 up, 8 in 2024-03-30T14:12:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:12:46.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:46.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T14:12:46.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:46 smithi138 ceph-mon[33397]: osdmap e723: 8 total, 8 up, 8 in 2024-03-30T14:12:46.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:12:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:12:46.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T14:12:46.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[29392]: osdmap e722: 8 total, 8 up, 8 in 2024-03-30T14:12:46.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:12:46.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:46.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T14:12:46.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[29392]: osdmap e723: 8 total, 8 up, 8 in 2024-03-30T14:12:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T14:12:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[37717]: osdmap e722: 8 total, 8 up, 8 in 2024-03-30T14:12:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T14:12:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T14:12:46.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:46 smithi102 ceph-mon[37717]: osdmap e723: 8 total, 8 up, 8 in 2024-03-30T14:12:47.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:47 smithi138 ceph-mon[33397]: pgmap v1114: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:47.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:47 smithi102 ceph-mon[29392]: pgmap v1114: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:47.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:47 smithi102 ceph-mon[37717]: pgmap v1114: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:12:48.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:48.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:48.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:49.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:49 smithi138 ceph-mon[33397]: pgmap v1116: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:12:49.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:12:49.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:49 smithi138 ceph-mon[33397]: osdmap e724: 8 total, 8 up, 8 in 2024-03-30T14:12:49.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:49.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[29392]: pgmap v1116: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:12:49.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:12:49.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[29392]: osdmap e724: 8 total, 8 up, 8 in 2024-03-30T14:12:49.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:49.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[37717]: pgmap v1116: 236 pgs: 236 active+clean; 4.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:12:49.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:12:49.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[37717]: osdmap e724: 8 total, 8 up, 8 in 2024-03-30T14:12:49.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:50.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]': finished 2024-03-30T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:50 smithi138 ceph-mon[33397]: osdmap e725: 8 total, 8 up, 8 in 2024-03-30T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:50.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]': finished 2024-03-30T14:12:50.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[29392]: osdmap e725: 8 total, 8 up, 8 in 2024-03-30T14:12:50.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:50.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:50.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]': finished 2024-03-30T14:12:50.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[37717]: osdmap e725: 8 total, 8 up, 8 in 2024-03-30T14:12:50.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:12:50.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-142"}]: dispatch 2024-03-30T14:12:51.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:51 smithi138 ceph-mon[33397]: pgmap v1119: 236 pgs: 236 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:12:51.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:51 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:51.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:51 smithi138 ceph-mon[33397]: osdmap e726: 8 total, 8 up, 8 in 2024-03-30T14:12:51.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:51 smithi102 ceph-mon[29392]: pgmap v1119: 236 pgs: 236 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:12:51.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:51 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:51.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:51 smithi102 ceph-mon[29392]: osdmap e726: 8 total, 8 up, 8 in 2024-03-30T14:12:51.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:51 smithi102 ceph-mon[37717]: pgmap v1119: 236 pgs: 236 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T14:12:51.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:51 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:51.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:51 smithi102 ceph-mon[37717]: osdmap e726: 8 total, 8 up, 8 in 2024-03-30T14:12:52.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:52 smithi138 ceph-mon[33397]: osdmap e727: 8 total, 8 up, 8 in 2024-03-30T14:12:52.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:52.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:52 smithi102 ceph-mon[29392]: osdmap e727: 8 total, 8 up, 8 in 2024-03-30T14:12:52.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:52.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:52 smithi102 ceph-mon[37717]: osdmap e727: 8 total, 8 up, 8 in 2024-03-30T14:12:52.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:12:53.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:53 smithi138 ceph-mon[33397]: pgmap v1122: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:12:53.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:53.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:53 smithi138 ceph-mon[33397]: osdmap e728: 8 total, 8 up, 8 in 2024-03-30T14:12:53.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:53.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[29392]: pgmap v1122: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:12:53.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:53.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[29392]: osdmap e728: 8 total, 8 up, 8 in 2024-03-30T14:12:53.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:53.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[37717]: pgmap v1122: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T14:12:53.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:12:53.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[37717]: osdmap e728: 8 total, 8 up, 8 in 2024-03-30T14:12:53.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:12:54.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:54.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:54 smithi138 ceph-mon[33397]: osdmap e729: 8 total, 8 up, 8 in 2024-03-30T14:12:54.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:12:54.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:54.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:54 smithi102 ceph-mon[29392]: osdmap e729: 8 total, 8 up, 8 in 2024-03-30T14:12:54.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:12:54.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:12:54.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:54 smithi102 ceph-mon[37717]: osdmap e729: 8 total, 8 up, 8 in 2024-03-30T14:12:54.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:12:55.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:55 smithi138 ceph-mon[33397]: pgmap v1125: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:55.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-144"}]': finished 2024-03-30T14:12:55.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:55 smithi138 ceph-mon[33397]: osdmap e730: 8 total, 8 up, 8 in 2024-03-30T14:12:55.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:55 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-144", "mode": "readproxy"}]: dispatch 2024-03-30T14:12:55.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[29392]: pgmap v1125: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:55.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-144"}]': finished 2024-03-30T14:12:55.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[29392]: osdmap e730: 8 total, 8 up, 8 in 2024-03-30T14:12:55.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-144", "mode": "readproxy"}]: dispatch 2024-03-30T14:12:55.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[37717]: pgmap v1125: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 666 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:12:55.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-144"}]': finished 2024-03-30T14:12:55.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[37717]: osdmap e730: 8 total, 8 up, 8 in 2024-03-30T14:12:55.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:55 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-144", "mode": "readproxy"}]: dispatch 2024-03-30T14:12:56.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:12:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:12:56.835 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:12:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-144", "mode": "readproxy"}]': finished 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[29392]: osdmap e731: 8 total, 8 up, 8 in 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-144", "mode": "readproxy"}]': finished 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[37717]: osdmap e731: 8 total, 8 up, 8 in 2024-03-30T14:12:56.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:56.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:56 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:12:56.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:56 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-144", "mode": "readproxy"}]': finished 2024-03-30T14:12:56.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:56 smithi138 ceph-mon[33397]: osdmap e731: 8 total, 8 up, 8 in 2024-03-30T14:12:56.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:12:57.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:57 smithi138 ceph-mon[33397]: pgmap v1128: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:12:57.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:57 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:57.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:57.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:57 smithi102 ceph-mon[29392]: pgmap v1128: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:12:57.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:57 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:57.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:57.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:57 smithi102 ceph-mon[37717]: pgmap v1128: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:12:57.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:57 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:12:57.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:12:59.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:12:59 smithi102 ceph-mon[29392]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 874 B/s rd, 174 B/s wr, 1 op/s 2024-03-30T14:12:59.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:12:59 smithi102 ceph-mon[37717]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 874 B/s rd, 174 B/s wr, 1 op/s 2024-03-30T14:13:00.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:12:59 smithi138 ceph-mon[33397]: pgmap v1129: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 874 B/s rd, 174 B/s wr, 1 op/s 2024-03-30T14:13:01.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:01 smithi102 ceph-mon[29392]: pgmap v1130: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 149 B/s wr, 1 op/s 2024-03-30T14:13:01.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:01 smithi102 ceph-mon[37717]: pgmap v1130: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 149 B/s wr, 1 op/s 2024-03-30T14:13:02.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:01 smithi138 ceph-mon[33397]: pgmap v1130: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 149 B/s wr, 1 op/s 2024-03-30T14:13:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:03 smithi138 ceph-mon[33397]: pgmap v1131: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-30T14:13:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:03 smithi102 ceph-mon[29392]: pgmap v1131: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-30T14:13:04.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:03 smithi102 ceph-mon[37717]: pgmap v1131: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1 op/s 2024-03-30T14:13:06.057 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:05 smithi138 ceph-mon[33397]: pgmap v1132: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 104 B/s wr, 1 op/s 2024-03-30T14:13:06.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:05 smithi102 ceph-mon[29392]: pgmap v1132: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 104 B/s wr, 1 op/s 2024-03-30T14:13:06.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:05 smithi102 ceph-mon[37717]: pgmap v1132: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 104 B/s wr, 1 op/s 2024-03-30T14:13:06.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:13:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:13:06.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:13:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:13:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:13:06.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:06 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:06.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:06.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:06 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:06.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:07.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:06 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:07.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:07 smithi138 ceph-mon[33397]: pgmap v1133: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:13:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:07 smithi138 ceph-mon[33397]: osdmap e732: 8 total, 8 up, 8 in 2024-03-30T14:13:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:07 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:13:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[37717]: pgmap v1133: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:13:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[37717]: osdmap e732: 8 total, 8 up, 8 in 2024-03-30T14:13:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:13:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[29392]: pgmap v1133: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:13:08.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:08.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[29392]: osdmap e732: 8 total, 8 up, 8 in 2024-03-30T14:13:08.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:07 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:13:09.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:08 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:13:09.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]': finished 2024-03-30T14:13:09.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:08 smithi138 ceph-mon[33397]: osdmap e733: 8 total, 8 up, 8 in 2024-03-30T14:13:09.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:09.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]': finished 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[29392]: osdmap e733: 8 total, 8 up, 8 in 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]': finished 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[37717]: osdmap e733: 8 total, 8 up, 8 in 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:09.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-144"}]: dispatch 2024-03-30T14:13:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:09 smithi138 ceph-mon[33397]: pgmap v1136: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:13:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:09 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:09 smithi138 ceph-mon[33397]: osdmap e734: 8 total, 8 up, 8 in 2024-03-30T14:13:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:09 smithi102 ceph-mon[37717]: pgmap v1136: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:13:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:09 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:09 smithi102 ceph-mon[37717]: osdmap e734: 8 total, 8 up, 8 in 2024-03-30T14:13:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:09 smithi102 ceph-mon[29392]: pgmap v1136: 236 pgs: 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:13:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:09 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:10.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:09 smithi102 ceph-mon[29392]: osdmap e734: 8 total, 8 up, 8 in 2024-03-30T14:13:11.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:10 smithi138 ceph-mon[33397]: osdmap e735: 8 total, 8 up, 8 in 2024-03-30T14:13:11.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:11.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:10 smithi102 ceph-mon[29392]: osdmap e735: 8 total, 8 up, 8 in 2024-03-30T14:13:11.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:10 smithi102 ceph-mon[37717]: osdmap e735: 8 total, 8 up, 8 in 2024-03-30T14:13:11.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:11 smithi138 ceph-mon[33397]: pgmap v1139: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:11 smithi138 ceph-mon[33397]: osdmap e736: 8 total, 8 up, 8 in 2024-03-30T14:13:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:11 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[29392]: pgmap v1139: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[29392]: osdmap e736: 8 total, 8 up, 8 in 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[37717]: pgmap v1139: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[37717]: osdmap e736: 8 total, 8 up, 8 in 2024-03-30T14:13:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:11 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:13:13.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:13:13.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:12 smithi138 ceph-mon[33397]: osdmap e737: 8 total, 8 up, 8 in 2024-03-30T14:13:13.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:12 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:13.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:12 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:13.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[29392]: osdmap e737: 8 total, 8 up, 8 in 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[37717]: osdmap e737: 8 total, 8 up, 8 in 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:13.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:13.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:13 smithi138 ceph-mon[33397]: pgmap v1142: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-146"}]': finished 2024-03-30T14:13:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:13 smithi138 ceph-mon[33397]: osdmap e738: 8 total, 8 up, 8 in 2024-03-30T14:13:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-146", "mode": "writeback"}]: dispatch 2024-03-30T14:13:14.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:13 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:13:14.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-146", "mode": "writeback"}]': finished 2024-03-30T14:13:14.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:13 smithi138 ceph-mon[33397]: osdmap e739: 8 total, 8 up, 8 in 2024-03-30T14:13:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[29392]: pgmap v1142: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-146"}]': finished 2024-03-30T14:13:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[29392]: osdmap e738: 8 total, 8 up, 8 in 2024-03-30T14:13:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-146", "mode": "writeback"}]: dispatch 2024-03-30T14:13:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:13:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-146", "mode": "writeback"}]': finished 2024-03-30T14:13:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[29392]: osdmap e739: 8 total, 8 up, 8 in 2024-03-30T14:13:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[37717]: pgmap v1142: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi102-75117-111", "overlaypool": "test-rados-api-smithi102-75117-146"}]': finished 2024-03-30T14:13:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[37717]: osdmap e738: 8 total, 8 up, 8 in 2024-03-30T14:13:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-146", "mode": "writeback"}]: dispatch 2024-03-30T14:13:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:13:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi102-75117-146", "mode": "writeback"}]': finished 2024-03-30T14:13:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:13 smithi102 ceph-mon[37717]: osdmap e739: 8 total, 8 up, 8 in 2024-03-30T14:13:15.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:13:15.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:13:15.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T14:13:16.068 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:15 smithi138 ceph-mon[33397]: pgmap v1145: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:16.068 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:13:16.068 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:15 smithi138 ceph-mon[33397]: osdmap e740: 8 total, 8 up, 8 in 2024-03-30T14:13:16.068 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:13:16.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[29392]: pgmap v1145: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:16.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:13:16.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[29392]: osdmap e740: 8 total, 8 up, 8 in 2024-03-30T14:13:16.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:13:16.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[37717]: pgmap v1145: 236 pgs: 32 creating+peering, 204 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:16.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_count","val": "2"}]': finished 2024-03-30T14:13:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[37717]: osdmap e740: 8 total, 8 up, 8 in 2024-03-30T14:13:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T14:13:16.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:13:16.783 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:13:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:13:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:13:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: osdmap e741: 8 total, 8 up, 8 in 2024-03-30T14:13:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:13:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:13:17.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:13:17.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: osdmap e742: 8 total, 8 up, 8 in 2024-03-30T14:13:17.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:13:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:13:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: osdmap e741: 8 total, 8 up, 8 in 2024-03-30T14:13:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:13:17.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: osdmap e742: 8 total, 8 up, 8 in 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_period","val": "600"}]': finished 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: osdmap e741: 8 total, 8 up, 8 in 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:17.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:13:17.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T14:13:17.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: osdmap e742: 8 total, 8 up, 8 in 2024-03-30T14:13:17.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T14:13:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:17 smithi138 ceph-mon[33397]: pgmap v1148: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:13:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:13:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:17 smithi138 ceph-mon[33397]: osdmap e743: 8 total, 8 up, 8 in 2024-03-30T14:13:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[29392]: pgmap v1148: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[29392]: osdmap e743: 8 total, 8 up, 8 in 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[37717]: pgmap v1148: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[37717]: osdmap e743: 8 total, 8 up, 8 in 2024-03-30T14:13:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T14:13:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:19 smithi138 ceph-mon[33397]: pgmap v1151: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:13:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:19 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "target_max_objects","val": "1"}]': finished 2024-03-30T14:13:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:19 smithi138 ceph-mon[33397]: osdmap e744: 8 total, 8 up, 8 in 2024-03-30T14:13:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:19 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:13:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:19 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:19 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[29392]: pgmap v1151: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "target_max_objects","val": "1"}]': finished 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[29392]: osdmap e744: 8 total, 8 up, 8 in 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[37717]: pgmap v1151: 236 pgs: 236 active+clean; 4.4 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-146","var": "target_max_objects","val": "1"}]': finished 2024-03-30T14:13:20.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[37717]: osdmap e744: 8 total, 8 up, 8 in 2024-03-30T14:13:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:13:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:19 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:20.680 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:13:20 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=sqlstore.transactions t=2024-03-30T14:13:20.420234414Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T14:13:21.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:20 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:13:21.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:20 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:13:21.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:20 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:21.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:20 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:13:21.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:20 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:13:21.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:20 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:21.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:20 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:13:21.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:20 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:13:21.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:20 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:21 smithi138 ceph-mon[33397]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:21 smithi138 ceph-mon[33397]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T14:13:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:21 smithi102 ceph-mon[29392]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:21 smithi102 ceph-mon[29392]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T14:13:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:21 smithi102 ceph-mon[37717]: pgmap v1153: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:21 smithi102 ceph-mon[37717]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T14:13:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:23 smithi138 ceph-mon[33397]: pgmap v1154: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:24.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:23 smithi102 ceph-mon[29392]: pgmap v1154: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:24.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:23 smithi102 ceph-mon[37717]: pgmap v1154: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:26.076 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:25 smithi138 ceph-mon[33397]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 0 op/s 2024-03-30T14:13:26.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:25 smithi102 ceph-mon[29392]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 0 op/s 2024-03-30T14:13:26.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:25 smithi102 ceph-mon[37717]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 0 op/s 2024-03-30T14:13:26.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:13:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:13:26.794 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:13:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:13:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:13:27.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:27.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:27.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:27 smithi138 ceph-mon[33397]: pgmap v1156: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:27 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[29392]: pgmap v1156: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[37717]: pgmap v1156: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:28.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]: dispatch 2024-03-30T14:13:28.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:27 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]: dispatch 2024-03-30T14:13:29.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:29 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]': finished 2024-03-30T14:13:29.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:29 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]': finished 2024-03-30T14:13:29.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:29 smithi138 ceph-mon[33397]: osdmap e745: 8 total, 8 up, 8 in 2024-03-30T14:13:29.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:29 smithi138 ceph-mon[33397]: pgmap v1158: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:29.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]': finished 2024-03-30T14:13:29.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]': finished 2024-03-30T14:13:29.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[29392]: osdmap e745: 8 total, 8 up, 8 in 2024-03-30T14:13:29.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[29392]: pgmap v1158: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:29.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.c", "id": [1, 0]}]': finished 2024-03-30T14:13:29.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.1f", "id": [1, 0]}]': finished 2024-03-30T14:13:29.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[37717]: osdmap e745: 8 total, 8 up, 8 in 2024-03-30T14:13:29.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:29 smithi102 ceph-mon[37717]: pgmap v1158: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T14:13:30.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:30 smithi138 ceph-mon[33397]: osdmap e746: 8 total, 8 up, 8 in 2024-03-30T14:13:30.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:30.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:30 smithi102 ceph-mon[29392]: osdmap e746: 8 total, 8 up, 8 in 2024-03-30T14:13:30.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:30.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:30 smithi102 ceph-mon[37717]: osdmap e746: 8 total, 8 up, 8 in 2024-03-30T14:13:30.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:31 smithi138 ceph-mon[33397]: osdmap e747: 8 total, 8 up, 8 in 2024-03-30T14:13:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:31 smithi138 ceph-mon[33397]: pgmap v1161: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T14:13:31.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]': finished 2024-03-30T14:13:31.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:31 smithi138 ceph-mon[33397]: osdmap e748: 8 total, 8 up, 8 in 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[29392]: osdmap e747: 8 total, 8 up, 8 in 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[29392]: pgmap v1161: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]': finished 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[29392]: osdmap e748: 8 total, 8 up, 8 in 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[37717]: osdmap e747: 8 total, 8 up, 8 in 2024-03-30T14:13:31.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:31.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[37717]: pgmap v1161: 236 pgs: 236 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T14:13:31.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]': finished 2024-03-30T14:13:31.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:31 smithi102 ceph-mon[37717]: osdmap e748: 8 total, 8 up, 8 in 2024-03-30T14:13:33.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:33.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:33.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:33.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:33.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:33.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-146"}]: dispatch 2024-03-30T14:13:34.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:33 smithi138 ceph-mon[33397]: osdmap e749: 8 total, 8 up, 8 in 2024-03-30T14:13:34.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:33 smithi138 ceph-mon[33397]: pgmap v1164: 204 pgs: 204 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:13:34.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:33 smithi138 ceph-mon[33397]: osdmap e750: 8 total, 8 up, 8 in 2024-03-30T14:13:34.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[29392]: osdmap e749: 8 total, 8 up, 8 in 2024-03-30T14:13:34.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[29392]: pgmap v1164: 204 pgs: 204 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:13:34.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[29392]: osdmap e750: 8 total, 8 up, 8 in 2024-03-30T14:13:34.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:34.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[37717]: osdmap e749: 8 total, 8 up, 8 in 2024-03-30T14:13:34.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[37717]: pgmap v1164: 204 pgs: 204 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T14:13:34.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[37717]: osdmap e750: 8 total, 8 up, 8 in 2024-03-30T14:13:34.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:35.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:34 smithi138 ceph-mon[33397]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T14:13:35.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:35.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:34 smithi138 ceph-mon[33397]: osdmap e751: 8 total, 8 up, 8 in 2024-03-30T14:13:35.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[29392]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[29392]: osdmap e751: 8 total, 8 up, 8 in 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[37717]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[37717]: osdmap e751: 8 total, 8 up, 8 in 2024-03-30T14:13:35.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:13:36.088 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:35 smithi138 ceph-mon[33397]: pgmap v1167: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:36.088 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:13:36.088 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:35 smithi138 ceph-mon[33397]: osdmap e752: 8 total, 8 up, 8 in 2024-03-30T14:13:36.088 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]: dispatch 2024-03-30T14:13:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[29392]: pgmap v1167: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:13:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[29392]: osdmap e752: 8 total, 8 up, 8 in 2024-03-30T14:13:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]: dispatch 2024-03-30T14:13:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[37717]: pgmap v1167: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:36.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:13:36.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[37717]: osdmap e752: 8 total, 8 up, 8 in 2024-03-30T14:13:36.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]: dispatch 2024-03-30T14:13:36.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:13:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:13:36.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:13:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:13:36.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:36.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]': finished 2024-03-30T14:13:36.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[29392]: osdmap e753: 8 total, 8 up, 8 in 2024-03-30T14:13:36.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:36.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]: dispatch 2024-03-30T14:13:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]': finished 2024-03-30T14:13:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[37717]: osdmap e753: 8 total, 8 up, 8 in 2024-03-30T14:13:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:36.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]: dispatch 2024-03-30T14:13:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]': finished 2024-03-30T14:13:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:36 smithi138 ceph-mon[33397]: osdmap e753: 8 total, 8 up, 8 in 2024-03-30T14:13:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-148"}]: dispatch 2024-03-30T14:13:38.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:37 smithi138 ceph-mon[33397]: pgmap v1170: 236 pgs: 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:13:38.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:37 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:37 smithi138 ceph-mon[33397]: osdmap e754: 8 total, 8 up, 8 in 2024-03-30T14:13:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:37 smithi102 ceph-mon[29392]: pgmap v1170: 236 pgs: 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:13:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:37 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:37 smithi102 ceph-mon[29392]: osdmap e754: 8 total, 8 up, 8 in 2024-03-30T14:13:38.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:37 smithi102 ceph-mon[37717]: pgmap v1170: 236 pgs: 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T14:13:38.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:37 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:38.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:37 smithi102 ceph-mon[37717]: osdmap e754: 8 total, 8 up, 8 in 2024-03-30T14:13:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:39 smithi138 ceph-mon[33397]: osdmap e755: 8 total, 8 up, 8 in 2024-03-30T14:13:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:39 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:39 smithi138 ceph-mon[33397]: pgmap v1173: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:13:39.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:39 smithi102 ceph-mon[29392]: osdmap e755: 8 total, 8 up, 8 in 2024-03-30T14:13:39.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:39 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:39.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:39 smithi102 ceph-mon[29392]: pgmap v1173: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:13:39.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:39 smithi102 ceph-mon[37717]: osdmap e755: 8 total, 8 up, 8 in 2024-03-30T14:13:39.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:39 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:39.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:39 smithi102 ceph-mon[37717]: pgmap v1173: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:13:41.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:41.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:40 smithi138 ceph-mon[33397]: osdmap e756: 8 total, 8 up, 8 in 2024-03-30T14:13:41.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:41.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:40 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-150"}]: dispatch 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[29392]: osdmap e756: 8 total, 8 up, 8 in 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-150"}]: dispatch 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[37717]: osdmap e756: 8 total, 8 up, 8 in 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:41.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:40 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-150"}]: dispatch 2024-03-30T14:13:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:41 smithi138 ceph-mon[33397]: osdmap e757: 8 total, 8 up, 8 in 2024-03-30T14:13:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:41 smithi138 ceph-mon[33397]: pgmap v1176: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:13:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:41 smithi138 ceph-mon[33397]: osdmap e758: 8 total, 8 up, 8 in 2024-03-30T14:13:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:41 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:42.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[37717]: osdmap e757: 8 total, 8 up, 8 in 2024-03-30T14:13:42.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[37717]: pgmap v1176: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:13:42.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[37717]: osdmap e758: 8 total, 8 up, 8 in 2024-03-30T14:13:42.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[29392]: osdmap e757: 8 total, 8 up, 8 in 2024-03-30T14:13:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[29392]: pgmap v1176: 204 pgs: 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T14:13:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[29392]: osdmap e758: 8 total, 8 up, 8 in 2024-03-30T14:13:42.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:41 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:43.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:43 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:43.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:43.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:43.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:43 smithi138 ceph-mon[33397]: osdmap e759: 8 total, 8 up, 8 in 2024-03-30T14:13:43.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:43 smithi138 ceph-mon[33397]: pgmap v1179: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:13:43.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:43.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:43 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-152"}]: dispatch 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[37717]: osdmap e759: 8 total, 8 up, 8 in 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[37717]: pgmap v1179: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-152"}]: dispatch 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:13:43.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:43.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:43.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[29392]: osdmap e759: 8 total, 8 up, 8 in 2024-03-30T14:13:43.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[29392]: pgmap v1179: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:13:43.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:43.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:43 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-152"}]: dispatch 2024-03-30T14:13:44.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:44 smithi138 ceph-mon[33397]: osdmap e760: 8 total, 8 up, 8 in 2024-03-30T14:13:44.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:44 smithi102 ceph-mon[29392]: osdmap e760: 8 total, 8 up, 8 in 2024-03-30T14:13:44.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:44 smithi102 ceph-mon[37717]: osdmap e760: 8 total, 8 up, 8 in 2024-03-30T14:13:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:45 smithi138 ceph-mon[33397]: osdmap e761: 8 total, 8 up, 8 in 2024-03-30T14:13:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:45 smithi138 ceph-mon[33397]: pgmap v1182: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:45 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:45 smithi138 ceph-mon[33397]: osdmap e762: 8 total, 8 up, 8 in 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[37717]: osdmap e761: 8 total, 8 up, 8 in 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[37717]: pgmap v1182: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[37717]: osdmap e762: 8 total, 8 up, 8 in 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[29392]: osdmap e761: 8 total, 8 up, 8 in 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[29392]: pgmap v1182: 236 pgs: 32 unknown, 204 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi102-75117-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:13:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:45 smithi102 ceph-mon[29392]: osdmap e762: 8 total, 8 up, 8 in 2024-03-30T14:13:46.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:13:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:13:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:46 smithi138 ceph-mon[33397]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-111","var": "dedup_tier","val": "test-rados-api-smithi102-75117-154"}]: dispatch 2024-03-30T14:13:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:46 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-154"}]: dispatch 2024-03-30T14:13:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:46 smithi138 ceph-mon[33397]: osdmap e763: 8 total, 8 up, 8 in 2024-03-30T14:13:46.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:13:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:13:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[37717]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-111","var": "dedup_tier","val": "test-rados-api-smithi102-75117-154"}]: dispatch 2024-03-30T14:13:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-154"}]: dispatch 2024-03-30T14:13:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[37717]: osdmap e763: 8 total, 8 up, 8 in 2024-03-30T14:13:46.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[29392]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:46.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi102-75117-111","var": "dedup_tier","val": "test-rados-api-smithi102-75117-154"}]: dispatch 2024-03-30T14:13:46.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:46.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi102-75117-111", "tierpool": "test-rados-api-smithi102-75117-154"}]: dispatch 2024-03-30T14:13:46.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:46.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:46 smithi102 ceph-mon[29392]: osdmap e763: 8 total, 8 up, 8 in 2024-03-30T14:13:47.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:47 smithi138 ceph-mon[33397]: pgmap v1185: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T14:13:47.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:47 smithi138 ceph-mon[33397]: osdmap e764: 8 total, 8 up, 8 in 2024-03-30T14:13:47.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:47 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:47.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:47 smithi102 ceph-mon[37717]: pgmap v1185: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T14:13:47.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:47 smithi102 ceph-mon[37717]: osdmap e764: 8 total, 8 up, 8 in 2024-03-30T14:13:47.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:47 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:47.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:47 smithi102 ceph-mon[29392]: pgmap v1185: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T14:13:47.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:47 smithi102 ceph-mon[29392]: osdmap e764: 8 total, 8 up, 8 in 2024-03-30T14:13:47.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:47 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7788 ms) 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13059 ms) 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8041 ms) 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13078 ms) 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7857 ms) 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8049 ms) 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-30T14:13:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: ok, hit_set contains 324:602f83fe:::foo:head 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8585 ms) 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: first is 1711807943 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,1711807948,1711807949,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,1711807948,1711807949,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,1711807948,1711807949,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,1711807948,1711807949,1711807951,1711807952,0 2024-03-30T14:13:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,1711807948,1711807949,1711807951,1711807952,0 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807943,1711807945,1711807946,1711807948,1711807949,1711807951,1711807952,0 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: got ls 1711807946,1711807948,1711807949,1711807951,1711807952,1711807954,1711807955,0 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: first now 1711807946, trimmed 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20222 ms) 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: foo0 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (15060 ms) 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18076 ms) 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23725 ms) 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-30T14:13:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5018 ms) 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3009 ms) 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3009 ms) 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3021 ms) 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (232314 ms total) 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (866173 ms total) 2024-03-30T14:13:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-30T14:13:49.529 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74777 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74777 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74923 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74923 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75535 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75535 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75243 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75243 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75037 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75037 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74857 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74857 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75168 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75168 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75682 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75682 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75710 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75710 2024-03-30T14:13:49.531 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74875 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74875 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75555 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75555 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75402 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75402 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74752 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74752 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74815 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74815 2024-03-30T14:13:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75497 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75497 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74743 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74743 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74763 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74763 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75805 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75805 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74938 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74938 2024-03-30T14:13:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:13:49.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75848 2024-03-30T14:13:49.534 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75848 2024-03-30T14:13:49.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:49.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:49 smithi138 ceph-mon[33397]: osdmap e765: 8 total, 8 up, 8 in 2024-03-30T14:13:49.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:49.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:49 smithi138 ceph-mon[33397]: pgmap v1188: 196 pgs: 196 active+clean; 582 KiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[37717]: osdmap e765: 8 total, 8 up, 8 in 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[37717]: pgmap v1188: 196 pgs: 196 active+clean; 582 KiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[29392]: osdmap e765: 8 total, 8 up, 8 in 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]: dispatch 2024-03-30T14:13:49.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:49 smithi102 ceph-mon[29392]: pgmap v1188: 196 pgs: 196 active+clean; 582 KiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:50.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:50.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:50 smithi138 ceph-mon[33397]: osdmap e766: 8 total, 8 up, 8 in 2024-03-30T14:13:50.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:50.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:50 smithi102 ceph-mon[37717]: osdmap e766: 8 total, 8 up, 8 in 2024-03-30T14:13:50.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2126282101' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi102-75117-111"}]': finished 2024-03-30T14:13:50.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:50 smithi102 ceph-mon[29392]: osdmap e766: 8 total, 8 up, 8 in 2024-03-30T14:13:51.832 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:51 smithi138 ceph-mon[33397]: pgmap v1190: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:51.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:51 smithi102 ceph-mon[29392]: pgmap v1190: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:51.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:51 smithi102 ceph-mon[37717]: pgmap v1190: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:52.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:52 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:52.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:52 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:52.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:52 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:13:53.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:53 smithi138 ceph-mon[33397]: pgmap v1191: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:53.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:53 smithi102 ceph-mon[29392]: pgmap v1191: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:53.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:53 smithi102 ceph-mon[37717]: pgmap v1191: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:13:55.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:55 smithi138 ceph-mon[33397]: pgmap v1192: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-30T14:13:55.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:55 smithi102 ceph-mon[29392]: pgmap v1192: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-30T14:13:55.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:55 smithi102 ceph-mon[37717]: pgmap v1192: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-30T14:13:56.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:13:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:13:56.835 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:56.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:56.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:13:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:13:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:13:56.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:13:57.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:57 smithi138 ceph-mon[33397]: pgmap v1193: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:57.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:57.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:57 smithi102 ceph-mon[29392]: pgmap v1193: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:57.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:57 smithi102 ceph-mon[37717]: pgmap v1193: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:13:59.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:13:59 smithi138 ceph-mon[33397]: pgmap v1194: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:13:59.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:13:59 smithi102 ceph-mon[29392]: pgmap v1194: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:13:59.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:13:59 smithi102 ceph-mon[37717]: pgmap v1194: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:14:01.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:01 smithi138 ceph-mon[33397]: pgmap v1195: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T14:14:01.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:01 smithi102 ceph-mon[29392]: pgmap v1195: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T14:14:01.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:01 smithi102 ceph-mon[37717]: pgmap v1195: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T14:14:03.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:03 smithi138 ceph-mon[33397]: pgmap v1196: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:03.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:03 smithi102 ceph-mon[29392]: pgmap v1196: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:03.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:03 smithi102 ceph-mon[37717]: pgmap v1196: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:05.931 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:05 smithi138 ceph-mon[33397]: pgmap v1197: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:05.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:05 smithi102 ceph-mon[29392]: pgmap v1197: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:05.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:05 smithi102 ceph-mon[37717]: pgmap v1197: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:06.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:14:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:14:06.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:06.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:06.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:06.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:14:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:14:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:14:07.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:07 smithi138 ceph-mon[33397]: pgmap v1198: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:07.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:07 smithi102 ceph-mon[29392]: pgmap v1198: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:07.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:07 smithi102 ceph-mon[37717]: pgmap v1198: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:09.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:09 smithi138 ceph-mon[33397]: pgmap v1199: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:09.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:09 smithi102 ceph-mon[29392]: pgmap v1199: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:09.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:09 smithi102 ceph-mon[37717]: pgmap v1199: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:11.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:11 smithi138 ceph-mon[33397]: pgmap v1200: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:11.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:11 smithi102 ceph-mon[29392]: pgmap v1200: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:11.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:11 smithi102 ceph-mon[37717]: pgmap v1200: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:12.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:12.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:12.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:13.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:13 smithi138 ceph-mon[33397]: pgmap v1201: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:13.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:13 smithi102 ceph-mon[29392]: pgmap v1201: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:13.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:13 smithi102 ceph-mon[37717]: pgmap v1201: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:15.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:15 smithi138 ceph-mon[33397]: pgmap v1202: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:15.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:15 smithi102 ceph-mon[29392]: pgmap v1202: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:15.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:15 smithi102 ceph-mon[37717]: pgmap v1202: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:16.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:14:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:14:16.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:16.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:16.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:16.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:14:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:14:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:14:17.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:17 smithi138 ceph-mon[33397]: pgmap v1203: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:17.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:17 smithi102 ceph-mon[29392]: pgmap v1203: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:17.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:17 smithi102 ceph-mon[37717]: pgmap v1203: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:19.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:19 smithi138 ceph-mon[33397]: pgmap v1204: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:19.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:19 smithi102 ceph-mon[29392]: pgmap v1204: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:19.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:19 smithi102 ceph-mon[37717]: pgmap v1204: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:20.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:20 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:14:20.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:20 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:14:20.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:20 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:14:21.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:21 smithi138 ceph-mon[33397]: pgmap v1205: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:21.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:21 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:14:21.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:21 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:14:21.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:21 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:14:21.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[29392]: pgmap v1205: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:21.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:14:21.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:14:21.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:14:21.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[37717]: pgmap v1205: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:21.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:14:21.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:14:21.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:21 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:14:23.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:23 smithi138 ceph-mon[33397]: pgmap v1206: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:23.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:23 smithi102 ceph-mon[29392]: pgmap v1206: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:23.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:23 smithi102 ceph-mon[37717]: pgmap v1206: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:25.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:25 smithi138 ceph-mon[33397]: pgmap v1207: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:25.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:25 smithi102 ceph-mon[29392]: pgmap v1207: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:25.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:25 smithi102 ceph-mon[37717]: pgmap v1207: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:26.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:14:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:14:26.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:26.836 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:26.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:14:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:14:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:14:26.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:27.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:27 smithi138 ceph-mon[33397]: pgmap v1208: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:27.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:27.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:27 smithi102 ceph-mon[29392]: pgmap v1208: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:27.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:27.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:27 smithi102 ceph-mon[37717]: pgmap v1208: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:27.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:29 smithi138 ceph-mon[33397]: pgmap v1209: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:29.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:29 smithi102 ceph-mon[29392]: pgmap v1209: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:29 smithi102 ceph-mon[37717]: pgmap v1209: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:31 smithi138 ceph-mon[33397]: pgmap v1210: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:31.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:31 smithi102 ceph-mon[29392]: pgmap v1210: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:31.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:31 smithi102 ceph-mon[37717]: pgmap v1210: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:33.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:33 smithi138 ceph-mon[33397]: pgmap v1211: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:33.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:33 smithi102 ceph-mon[29392]: pgmap v1211: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:33.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:33 smithi102 ceph-mon[37717]: pgmap v1211: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:35.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:35 smithi138 ceph-mon[33397]: pgmap v1212: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:35.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:35 smithi102 ceph-mon[29392]: pgmap v1212: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:35.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:35 smithi102 ceph-mon[37717]: pgmap v1212: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:36.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:14:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:14:36.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:36.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:36.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:36.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:14:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:14:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:14:37.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:37 smithi138 ceph-mon[33397]: pgmap v1213: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:37.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:37 smithi102 ceph-mon[37717]: pgmap v1213: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:37.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:37 smithi102 ceph-mon[29392]: pgmap v1213: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:39.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:39 smithi138 ceph-mon[33397]: pgmap v1214: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:39.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:39 smithi102 ceph-mon[29392]: pgmap v1214: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:39.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:39 smithi102 ceph-mon[37717]: pgmap v1214: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:41.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:41 smithi102 ceph-mon[37717]: pgmap v1215: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:41 smithi102 ceph-mon[29392]: pgmap v1215: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:41 smithi138 ceph-mon[33397]: pgmap v1215: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:43.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:43.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:43.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:44.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:43 smithi138 ceph-mon[33397]: pgmap v1216: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:43 smithi102 ceph-mon[37717]: pgmap v1216: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:44.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:43 smithi102 ceph-mon[29392]: pgmap v1216: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:46.162 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:45 smithi138 ceph-mon[33397]: pgmap v1217: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:46.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:45 smithi102 ceph-mon[37717]: pgmap v1217: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:46.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:45 smithi102 ceph-mon[29392]: pgmap v1217: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:46.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:14:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:14:46.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:46.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:14:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:14:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:14:47.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:47 smithi138 ceph-mon[33397]: pgmap v1218: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:47 smithi102 ceph-mon[29392]: pgmap v1218: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:48.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:47 smithi102 ceph-mon[37717]: pgmap v1218: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:50.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:49 smithi138 ceph-mon[33397]: pgmap v1219: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:50.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:49 smithi102 ceph-mon[29392]: pgmap v1219: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:50.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:49 smithi102 ceph-mon[37717]: pgmap v1219: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:51.982 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:51 smithi138 ceph-mon[33397]: pgmap v1220: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:52.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:51 smithi102 ceph-mon[29392]: pgmap v1220: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:52.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:51 smithi102 ceph-mon[37717]: pgmap v1220: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:54.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:53 smithi138 ceph-mon[33397]: pgmap v1221: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:53 smithi102 ceph-mon[29392]: pgmap v1221: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:54.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:53 smithi102 ceph-mon[37717]: pgmap v1221: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:56.174 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:55 smithi138 ceph-mon[33397]: pgmap v1222: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:56.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:55 smithi102 ceph-mon[29392]: pgmap v1222: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:56.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:55 smithi102 ceph-mon[37717]: pgmap v1222: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:14:56.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:14:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:14:56.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:56.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:56.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:14:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:14:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:14:57.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:14:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:57 smithi138 ceph-mon[33397]: pgmap v1223: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:57 smithi102 ceph-mon[29392]: pgmap v1223: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:14:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:57 smithi102 ceph-mon[37717]: pgmap v1223: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:14:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:00.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:14:59 smithi138 ceph-mon[33397]: pgmap v1224: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:00.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:14:59 smithi102 ceph-mon[29392]: pgmap v1224: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:00.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:14:59 smithi102 ceph-mon[37717]: pgmap v1224: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:02.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:01 smithi138 ceph-mon[33397]: pgmap v1225: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:02.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:01 smithi102 ceph-mon[29392]: pgmap v1225: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:02.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:01 smithi102 ceph-mon[37717]: pgmap v1225: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:03 smithi138 ceph-mon[33397]: pgmap v1226: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:04.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:03 smithi102 ceph-mon[29392]: pgmap v1226: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:04.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:03 smithi102 ceph-mon[37717]: pgmap v1226: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:05 smithi138 ceph-mon[33397]: pgmap v1227: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:06.502 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:05 smithi102 ceph-mon[29392]: pgmap v1227: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:06.502 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:05 smithi102 ceph-mon[37717]: pgmap v1227: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:06.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:15:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:15:06.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:06.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:06.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:15:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:15:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:15:07.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:07 smithi138 ceph-mon[33397]: pgmap v1228: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:07 smithi102 ceph-mon[29392]: pgmap v1228: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:07 smithi102 ceph-mon[37717]: pgmap v1228: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:09 smithi138 ceph-mon[33397]: pgmap v1229: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:09 smithi102 ceph-mon[29392]: pgmap v1229: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:09 smithi102 ceph-mon[37717]: pgmap v1229: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:11 smithi138 ceph-mon[33397]: pgmap v1230: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:12.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:11 smithi102 ceph-mon[29392]: pgmap v1230: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:12.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:11 smithi102 ceph-mon[37717]: pgmap v1230: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:13.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:13.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:13.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:13 smithi138 ceph-mon[33397]: pgmap v1231: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:15:14.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:13 smithi102 ceph-mon[29392]: pgmap v1231: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:15:14.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:13 smithi102 ceph-mon[37717]: pgmap v1231: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:15:16.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:15 smithi138 ceph-mon[33397]: pgmap v1232: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:16.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:15 smithi102 ceph-mon[29392]: pgmap v1232: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:16.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:15 smithi102 ceph-mon[37717]: pgmap v1232: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:16.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:15:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:15:16.766 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:15:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:15:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:15:17.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:17.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:17.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:17 smithi138 ceph-mon[33397]: pgmap v1233: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:17 smithi102 ceph-mon[29392]: pgmap v1233: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:17 smithi102 ceph-mon[37717]: pgmap v1233: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:19 smithi138 ceph-mon[33397]: pgmap v1234: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:20.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:19 smithi102 ceph-mon[37717]: pgmap v1234: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:19 smithi102 ceph-mon[29392]: pgmap v1234: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:21 smithi138 ceph-mon[33397]: pgmap v1235: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:21 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:21 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:15:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:21 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:15:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:21 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:15:22.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[29392]: pgmap v1235: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[37717]: pgmap v1235: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:15:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:21 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:15:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:23 smithi138 ceph-mon[33397]: pgmap v1236: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:23 smithi102 ceph-mon[29392]: pgmap v1236: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:24.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:23 smithi102 ceph-mon[37717]: pgmap v1236: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:26.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:25 smithi138 ceph-mon[33397]: pgmap v1237: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:26.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:25 smithi102 ceph-mon[29392]: pgmap v1237: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:26.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:25 smithi102 ceph-mon[37717]: pgmap v1237: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:26.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:15:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:15:26.791 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:15:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:15:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:15:27.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:27.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:27.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:27 smithi138 ceph-mon[33397]: pgmap v1238: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:27 smithi102 ceph-mon[29392]: pgmap v1238: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:27 smithi102 ceph-mon[37717]: pgmap v1238: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:30.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:29 smithi138 ceph-mon[33397]: pgmap v1239: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:30.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:29 smithi102 ceph-mon[29392]: pgmap v1239: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:30.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:29 smithi102 ceph-mon[37717]: pgmap v1239: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:32.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:31 smithi138 ceph-mon[33397]: pgmap v1240: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:32.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:31 smithi102 ceph-mon[29392]: pgmap v1240: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:32.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:31 smithi102 ceph-mon[37717]: pgmap v1240: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:34.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:33 smithi138 ceph-mon[33397]: pgmap v1241: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:33 smithi102 ceph-mon[29392]: pgmap v1241: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:34.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:33 smithi102 ceph-mon[37717]: pgmap v1241: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:36.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:35 smithi138 ceph-mon[33397]: pgmap v1242: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:36.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:35 smithi102 ceph-mon[29392]: pgmap v1242: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:35 smithi102 ceph-mon[37717]: pgmap v1242: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:36.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:15:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:15:36.809 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:15:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:15:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:15:37.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:37.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:37.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:38.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:37 smithi138 ceph-mon[33397]: pgmap v1243: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:38.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:37 smithi102 ceph-mon[29392]: pgmap v1243: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:38.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:37 smithi102 ceph-mon[37717]: pgmap v1243: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:40.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:39 smithi138 ceph-mon[33397]: pgmap v1244: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:40.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:39 smithi102 ceph-mon[29392]: pgmap v1244: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:39 smithi102 ceph-mon[37717]: pgmap v1244: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:42.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:41 smithi138 ceph-mon[33397]: pgmap v1245: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:42.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:41 smithi102 ceph-mon[29392]: pgmap v1245: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:42.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:41 smithi102 ceph-mon[37717]: pgmap v1245: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:43.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:43.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:43.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:44.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:43 smithi138 ceph-mon[33397]: pgmap v1246: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:43 smithi102 ceph-mon[29392]: pgmap v1246: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:43 smithi102 ceph-mon[37717]: pgmap v1246: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:46.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:45 smithi138 ceph-mon[33397]: pgmap v1247: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:46.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:45 smithi102 ceph-mon[29392]: pgmap v1247: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:46.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:45 smithi102 ceph-mon[37717]: pgmap v1247: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:46.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:15:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:15:46.833 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:15:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:15:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:15:47.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:47.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:47.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:47 smithi138 ceph-mon[33397]: pgmap v1248: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:47 smithi102 ceph-mon[29392]: pgmap v1248: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:47 smithi102 ceph-mon[37717]: pgmap v1248: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:50.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:49 smithi138 ceph-mon[33397]: pgmap v1249: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:50.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:49 smithi102 ceph-mon[29392]: pgmap v1249: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:50.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:49 smithi102 ceph-mon[37717]: pgmap v1249: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:52.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:51 smithi138 ceph-mon[33397]: pgmap v1250: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:52.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:51 smithi102 ceph-mon[29392]: pgmap v1250: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:52.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:51 smithi102 ceph-mon[37717]: pgmap v1250: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:54.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:53 smithi138 ceph-mon[33397]: pgmap v1251: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:54.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:53 smithi102 ceph-mon[29392]: pgmap v1251: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:54.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:53 smithi102 ceph-mon[37717]: pgmap v1251: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:56.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:55 smithi138 ceph-mon[33397]: pgmap v1252: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:56.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:55 smithi102 ceph-mon[29392]: pgmap v1252: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:56.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:55 smithi102 ceph-mon[37717]: pgmap v1252: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:15:56.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:15:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:15:56.850 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:15:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:15:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:15:57.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:57.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:57.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:15:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:57 smithi138 ceph-mon[33397]: pgmap v1253: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:57 smithi102 ceph-mon[37717]: pgmap v1253: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:15:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:57 smithi102 ceph-mon[29392]: pgmap v1253: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:15:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:00.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:15:59 smithi138 ceph-mon[33397]: pgmap v1254: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:00.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:15:59 smithi102 ceph-mon[37717]: pgmap v1254: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:00.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:15:59 smithi102 ceph-mon[29392]: pgmap v1254: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:02.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:01 smithi138 ceph-mon[33397]: pgmap v1255: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:02.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:01 smithi102 ceph-mon[29392]: pgmap v1255: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:02.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:01 smithi102 ceph-mon[37717]: pgmap v1255: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:03 smithi138 ceph-mon[33397]: pgmap v1256: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:04.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:03 smithi102 ceph-mon[29392]: pgmap v1256: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:04.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:03 smithi102 ceph-mon[37717]: pgmap v1256: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:06.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:05 smithi138 ceph-mon[33397]: pgmap v1257: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:06.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:05 smithi102 ceph-mon[29392]: pgmap v1257: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:06.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:05 smithi102 ceph-mon[37717]: pgmap v1257: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:06.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:16:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:16:06.870 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:16:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:16:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:16:07.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:07.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:07.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:08.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:07 smithi138 ceph-mon[33397]: pgmap v1258: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:07 smithi102 ceph-mon[29392]: pgmap v1258: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:08.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:07 smithi102 ceph-mon[37717]: pgmap v1258: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:09 smithi138 ceph-mon[33397]: pgmap v1259: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:10.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:09 smithi102 ceph-mon[29392]: pgmap v1259: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:10.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:09 smithi102 ceph-mon[37717]: pgmap v1259: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:11 smithi138 ceph-mon[33397]: pgmap v1260: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:12.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:11 smithi102 ceph-mon[29392]: pgmap v1260: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:12.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:11 smithi102 ceph-mon[37717]: pgmap v1260: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:13.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:13.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:13.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:13 smithi138 ceph-mon[33397]: pgmap v1261: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:14.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:13 smithi102 ceph-mon[29392]: pgmap v1261: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:13 smithi102 ceph-mon[37717]: pgmap v1261: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:16.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:15 smithi138 ceph-mon[33397]: pgmap v1262: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:16.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:15 smithi102 ceph-mon[29392]: pgmap v1262: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:16.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:15 smithi102 ceph-mon[37717]: pgmap v1262: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:16.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:16:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:16:16.895 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:16:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:16:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:16:17.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:17.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:17.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:17 smithi138 ceph-mon[33397]: pgmap v1263: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:17 smithi102 ceph-mon[37717]: pgmap v1263: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:17 smithi102 ceph-mon[29392]: pgmap v1263: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:19 smithi138 ceph-mon[33397]: pgmap v1264: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:20.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:19 smithi102 ceph-mon[29392]: pgmap v1264: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:20.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:19 smithi102 ceph-mon[37717]: pgmap v1264: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:22.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:21 smithi138 ceph-mon[33397]: pgmap v1265: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:21 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:16:22.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:21 smithi102 ceph-mon[37717]: pgmap v1265: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:22.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:21 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:16:22.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:21 smithi102 ceph-mon[29392]: pgmap v1265: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:22.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:21 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:16:23.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:22 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:16:23.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:22 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:16:23.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:22 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:16:23.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:22 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:16:23.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:22 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:16:23.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:22 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:16:23.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:22 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:16:23.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:22 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:16:23.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:22 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:16:24.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:24 smithi138 ceph-mon[33397]: pgmap v1266: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:24.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:24 smithi102 ceph-mon[29392]: pgmap v1266: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:24.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:24 smithi102 ceph-mon[37717]: pgmap v1266: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:26.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:26 smithi138 ceph-mon[33397]: pgmap v1267: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:26.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:16:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:16:26.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:26 smithi102 ceph-mon[29392]: pgmap v1267: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:26.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:26 smithi102 ceph-mon[37717]: pgmap v1267: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:26.684 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:16:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:16:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:16:27.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:27.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:27.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:28.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:28 smithi138 ceph-mon[33397]: pgmap v1268: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:28.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:28.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:28 smithi102 ceph-mon[37717]: pgmap v1268: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:28.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:28.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:28 smithi102 ceph-mon[29392]: pgmap v1268: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:28.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:29.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:29 smithi138 ceph-mon[33397]: pgmap v1269: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:29.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:29 smithi102 ceph-mon[29392]: pgmap v1269: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:29.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:29 smithi102 ceph-mon[37717]: pgmap v1269: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:31.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:31 smithi138 ceph-mon[33397]: pgmap v1270: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:31.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:31 smithi102 ceph-mon[29392]: pgmap v1270: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:31.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:31 smithi102 ceph-mon[37717]: pgmap v1270: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:33.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:33 smithi138 ceph-mon[33397]: pgmap v1271: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:33.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:33 smithi102 ceph-mon[37717]: pgmap v1271: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:33.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:33 smithi102 ceph-mon[29392]: pgmap v1271: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:35.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:35 smithi138 ceph-mon[33397]: pgmap v1272: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:35.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:35 smithi102 ceph-mon[37717]: pgmap v1272: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:35.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:35 smithi102 ceph-mon[29392]: pgmap v1272: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:36.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:16:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:16:36.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:36.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:36.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:36.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:16:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:16:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:16:37.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:37 smithi138 ceph-mon[33397]: pgmap v1273: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:37.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:37 smithi102 ceph-mon[37717]: pgmap v1273: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:37.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:37 smithi102 ceph-mon[29392]: pgmap v1273: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:39.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:39 smithi138 ceph-mon[33397]: pgmap v1274: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:39.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:39 smithi102 ceph-mon[29392]: pgmap v1274: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:39.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:39 smithi102 ceph-mon[37717]: pgmap v1274: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:41.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:41 smithi138 ceph-mon[33397]: pgmap v1275: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:41 smithi102 ceph-mon[29392]: pgmap v1275: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:41.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:41 smithi102 ceph-mon[37717]: pgmap v1275: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:42.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:42.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:42.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:43.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:43 smithi138 ceph-mon[33397]: pgmap v1276: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:43.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:43 smithi102 ceph-mon[29392]: pgmap v1276: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:43.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:43 smithi102 ceph-mon[37717]: pgmap v1276: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:45.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:45 smithi138 ceph-mon[33397]: pgmap v1277: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:45.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:45 smithi102 ceph-mon[29392]: pgmap v1277: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:45.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:45 smithi102 ceph-mon[37717]: pgmap v1277: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:46.667 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:16:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:16:46.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:46.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:46.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:46.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:16:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:16:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:16:47.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:47 smithi138 ceph-mon[33397]: pgmap v1278: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:47.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:47 smithi102 ceph-mon[29392]: pgmap v1278: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:47.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:47 smithi102 ceph-mon[37717]: pgmap v1278: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:49.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:49 smithi138 ceph-mon[33397]: pgmap v1279: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:49.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:49 smithi102 ceph-mon[29392]: pgmap v1279: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:49.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:49 smithi102 ceph-mon[37717]: pgmap v1279: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:51.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:51 smithi138 ceph-mon[33397]: pgmap v1280: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:51.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:51 smithi102 ceph-mon[29392]: pgmap v1280: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:51.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:51 smithi102 ceph-mon[37717]: pgmap v1280: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:16:53.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:53 smithi138 ceph-mon[33397]: pgmap v1281: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:53.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:53 smithi102 ceph-mon[37717]: pgmap v1281: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:53.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:53 smithi102 ceph-mon[29392]: pgmap v1281: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:55.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:55 smithi102 ceph-mon[29392]: pgmap v1282: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:55.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:55 smithi102 ceph-mon[37717]: pgmap v1282: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:56.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:55 smithi138 ceph-mon[33397]: pgmap v1282: 196 pgs: 196 active+clean; 582 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:16:56.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:16:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:16:56.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:56.835 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:56.836 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:16:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:16:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:16:57.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:16:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:57 smithi138 ceph-mon[33397]: pgmap v1283: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:16:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:57 smithi102 ceph-mon[29392]: pgmap v1283: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:16:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:16:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:57 smithi102 ceph-mon[37717]: pgmap v1283: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:16:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:00.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:16:59 smithi138 ceph-mon[33397]: pgmap v1284: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:00.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:16:59 smithi102 ceph-mon[29392]: pgmap v1284: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:00.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:16:59 smithi102 ceph-mon[37717]: pgmap v1284: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:02.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:01 smithi138 ceph-mon[33397]: pgmap v1285: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:02.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:01 smithi102 ceph-mon[29392]: pgmap v1285: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:02.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:01 smithi102 ceph-mon[37717]: pgmap v1285: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:04.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:03 smithi138 ceph-mon[33397]: pgmap v1286: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:04.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:03 smithi102 ceph-mon[29392]: pgmap v1286: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:04.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:03 smithi102 ceph-mon[37717]: pgmap v1286: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:06.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:05 smithi138 ceph-mon[33397]: pgmap v1287: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:06.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:05 smithi102 ceph-mon[29392]: pgmap v1287: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:06.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:05 smithi102 ceph-mon[37717]: pgmap v1287: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:06.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:17:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:17:06.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:06 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:06.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:06 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:06.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:17:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:17:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:17:07.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:06 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:08.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:07 smithi138 ceph-mon[33397]: pgmap v1288: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:07 smithi102 ceph-mon[29392]: pgmap v1288: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:08.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:07 smithi102 ceph-mon[37717]: pgmap v1288: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:17:10.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:09 smithi138 ceph-mon[33397]: pgmap v1289: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:10.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:09 smithi102 ceph-mon[37717]: pgmap v1289: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:10.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:09 smithi102 ceph-mon[29392]: pgmap v1289: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:12.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:11 smithi138 ceph-mon[33397]: pgmap v1290: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:11 smithi102 ceph-mon[37717]: pgmap v1290: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:11 smithi102 ceph-mon[29392]: pgmap v1290: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:13.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:13.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:13.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:14.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:13 smithi138 ceph-mon[33397]: pgmap v1291: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:14.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:13 smithi102 ceph-mon[29392]: pgmap v1291: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:13 smithi102 ceph-mon[37717]: pgmap v1291: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:16.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:15 smithi138 ceph-mon[33397]: pgmap v1292: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:16.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:15 smithi102 ceph-mon[29392]: pgmap v1292: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:16.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:15 smithi102 ceph-mon[37717]: pgmap v1292: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:16.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:17:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:17:16.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:16 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:16.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:16 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:16.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:17:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:17:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:17:17.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:16 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:17 smithi138 ceph-mon[33397]: pgmap v1293: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:17 smithi102 ceph-mon[29392]: pgmap v1293: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:17 smithi102 ceph-mon[37717]: pgmap v1293: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:19 smithi138 ceph-mon[33397]: pgmap v1294: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:20.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:19 smithi102 ceph-mon[29392]: pgmap v1294: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:20.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:19 smithi102 ceph-mon[37717]: pgmap v1294: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:22.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:21 smithi138 ceph-mon[33397]: pgmap v1295: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:22.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:21 smithi102 ceph-mon[29392]: pgmap v1295: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:22.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:21 smithi102 ceph-mon[37717]: pgmap v1295: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:23.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:22 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:17:23.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:22 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:17:23.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:22 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:17:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:23 smithi138 ceph-mon[33397]: pgmap v1296: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:23 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:23 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:17:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:23 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:17:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[29392]: pgmap v1296: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:17:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:17:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[37717]: pgmap v1296: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:17:24.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:17:24.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:23 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:17:26.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:25 smithi138 ceph-mon[33397]: pgmap v1297: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:26.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:25 smithi102 ceph-mon[29392]: pgmap v1297: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:26.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:25 smithi102 ceph-mon[37717]: pgmap v1297: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:26.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:17:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:17:26.755 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:17:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:17:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:17:27.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:26 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:27.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:26 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:27.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:26 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:27 smithi138 ceph-mon[33397]: pgmap v1298: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:28.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:27 smithi102 ceph-mon[29392]: pgmap v1298: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:28.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:27 smithi102 ceph-mon[37717]: pgmap v1298: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:28.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:30.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:29 smithi138 ceph-mon[33397]: pgmap v1299: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:30.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:29 smithi102 ceph-mon[29392]: pgmap v1299: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:30.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:29 smithi102 ceph-mon[37717]: pgmap v1299: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:32.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:31 smithi138 ceph-mon[33397]: pgmap v1300: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:32.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:31 smithi102 ceph-mon[29392]: pgmap v1300: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:31 smithi102 ceph-mon[37717]: pgmap v1300: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:34.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:33 smithi138 ceph-mon[33397]: pgmap v1301: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:33 smithi102 ceph-mon[29392]: pgmap v1301: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:34.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:33 smithi102 ceph-mon[37717]: pgmap v1301: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:36.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:35 smithi138 ceph-mon[33397]: pgmap v1302: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:36.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:35 smithi102 ceph-mon[29392]: pgmap v1302: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:36.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:35 smithi102 ceph-mon[37717]: pgmap v1302: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:36.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:17:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:17:36.773 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:17:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:17:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:17:37.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:36 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:37.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:36 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:37.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:36 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:38.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:37 smithi138 ceph-mon[33397]: pgmap v1303: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:38.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:37 smithi102 ceph-mon[29392]: pgmap v1303: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:38.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:37 smithi102 ceph-mon[37717]: pgmap v1303: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:40.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:39 smithi138 ceph-mon[33397]: pgmap v1304: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:40.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:39 smithi102 ceph-mon[29392]: pgmap v1304: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:39 smithi102 ceph-mon[37717]: pgmap v1304: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:42.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:41 smithi138 ceph-mon[33397]: pgmap v1305: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:42.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:41 smithi102 ceph-mon[29392]: pgmap v1305: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:42.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:41 smithi102 ceph-mon[37717]: pgmap v1305: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:43.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:43.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:43.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:43 smithi138 ceph-mon[33397]: pgmap v1306: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:43 smithi102 ceph-mon[37717]: pgmap v1306: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:44.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:43 smithi102 ceph-mon[29392]: pgmap v1306: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:46.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:45 smithi138 ceph-mon[33397]: pgmap v1307: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:46.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:45 smithi102 ceph-mon[29392]: pgmap v1307: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:46.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:45 smithi102 ceph-mon[37717]: pgmap v1307: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:46.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:17:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:17:46.799 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:17:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:17:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:17:47.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:46 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:47.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:46 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:47.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:46 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:47 smithi138 ceph-mon[33397]: pgmap v1308: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:47 smithi102 ceph-mon[29392]: pgmap v1308: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:47 smithi102 ceph-mon[37717]: pgmap v1308: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:50.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:49 smithi138 ceph-mon[33397]: pgmap v1309: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:50.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:49 smithi102 ceph-mon[29392]: pgmap v1309: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:50.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:49 smithi102 ceph-mon[37717]: pgmap v1309: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:52.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:51 smithi138 ceph-mon[33397]: pgmap v1310: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:52.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:51 smithi102 ceph-mon[37717]: pgmap v1310: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:52.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:51 smithi102 ceph-mon[29392]: pgmap v1310: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:54.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:53 smithi138 ceph-mon[33397]: pgmap v1311: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:54.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:53 smithi102 ceph-mon[29392]: pgmap v1311: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:54.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:53 smithi102 ceph-mon[37717]: pgmap v1311: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:56.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:55 smithi138 ceph-mon[33397]: pgmap v1312: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:56.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:55 smithi102 ceph-mon[29392]: pgmap v1312: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:56.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:55 smithi102 ceph-mon[37717]: pgmap v1312: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:17:56.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:17:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:17:56.817 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:17:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:17:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:17:57.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:56 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:57.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:56 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:57.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:56 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:17:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:57 smithi138 ceph-mon[33397]: pgmap v1313: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:57 smithi102 ceph-mon[29392]: pgmap v1313: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:17:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:57 smithi102 ceph-mon[37717]: pgmap v1313: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:17:58.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:00.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:17:59 smithi138 ceph-mon[33397]: pgmap v1314: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:00.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:17:59 smithi102 ceph-mon[29392]: pgmap v1314: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:00.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:17:59 smithi102 ceph-mon[37717]: pgmap v1314: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:02.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:01 smithi138 ceph-mon[33397]: pgmap v1315: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:02.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:01 smithi102 ceph-mon[29392]: pgmap v1315: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:02.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:01 smithi102 ceph-mon[37717]: pgmap v1315: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:04.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:03 smithi138 ceph-mon[33397]: pgmap v1316: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:04.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:03 smithi102 ceph-mon[29392]: pgmap v1316: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:04.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:03 smithi102 ceph-mon[37717]: pgmap v1316: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:06.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:05 smithi138 ceph-mon[33397]: pgmap v1317: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:06.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:05 smithi102 ceph-mon[29392]: pgmap v1317: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:06.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:05 smithi102 ceph-mon[37717]: pgmap v1317: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:06.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:18:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:18:06.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:18:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:18:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:18:08.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:07 smithi138 ceph-mon[33397]: pgmap v1318: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:07 smithi102 ceph-mon[29392]: pgmap v1318: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:08.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:07 smithi102 ceph-mon[37717]: pgmap v1318: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:10.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:09 smithi138 ceph-mon[33397]: pgmap v1319: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:10.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:09 smithi102 ceph-mon[29392]: pgmap v1319: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:10.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:09 smithi102 ceph-mon[37717]: pgmap v1319: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:12.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:11 smithi138 ceph-mon[33397]: pgmap v1320: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:12.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:11 smithi102 ceph-mon[29392]: pgmap v1320: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:12.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:11 smithi102 ceph-mon[37717]: pgmap v1320: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:13.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:13.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:13.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:14.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:13 smithi138 ceph-mon[33397]: pgmap v1321: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:14.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:13 smithi102 ceph-mon[29392]: pgmap v1321: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:13 smithi102 ceph-mon[37717]: pgmap v1321: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:16.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:15 smithi138 ceph-mon[33397]: pgmap v1322: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:16.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:15 smithi102 ceph-mon[29392]: pgmap v1322: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:16.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:15 smithi102 ceph-mon[37717]: pgmap v1322: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:16.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:18:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:18:16.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:18:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:18:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:18:18.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:18.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:17 smithi138 ceph-mon[33397]: pgmap v1323: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:17 smithi102 ceph-mon[29392]: pgmap v1323: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:17 smithi102 ceph-mon[37717]: pgmap v1323: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:20.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:19 smithi138 ceph-mon[33397]: pgmap v1324: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:18:20.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:19 smithi102 ceph-mon[29392]: pgmap v1324: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:18:20.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:19 smithi102 ceph-mon[37717]: pgmap v1324: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:18:21.429 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:18:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=cleanup t=2024-03-30T14:18:21.035729863Z level=info msg="Completed cleanup jobs" duration=6.491649ms 2024-03-30T14:18:22.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:21 smithi138 ceph-mon[33397]: pgmap v1325: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:22.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:21 smithi102 ceph-mon[29392]: pgmap v1325: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:22.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:21 smithi102 ceph-mon[37717]: pgmap v1325: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:24.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:23 smithi138 ceph-mon[33397]: pgmap v1326: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:24.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:23 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:18:24.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:23 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:24.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:23 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:18:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:23 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[37717]: pgmap v1326: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[29392]: pgmap v1326: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:18:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:18:24.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:23 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:18:26.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:25 smithi138 ceph-mon[33397]: pgmap v1327: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:26.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:25 smithi102 ceph-mon[29392]: pgmap v1327: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:26.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:25 smithi102 ceph-mon[37717]: pgmap v1327: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:26.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:18:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:18:26.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:18:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:18:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:18:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:27 smithi138 ceph-mon[33397]: pgmap v1328: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:27 smithi102 ceph-mon[29392]: pgmap v1328: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:27 smithi102 ceph-mon[37717]: pgmap v1328: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:30.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:29 smithi138 ceph-mon[33397]: pgmap v1329: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:30.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:29 smithi102 ceph-mon[29392]: pgmap v1329: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:30.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:29 smithi102 ceph-mon[37717]: pgmap v1329: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:32.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:31 smithi138 ceph-mon[33397]: pgmap v1330: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:32.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:31 smithi102 ceph-mon[29392]: pgmap v1330: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:31 smithi102 ceph-mon[37717]: pgmap v1330: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:34.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:33 smithi138 ceph-mon[33397]: pgmap v1331: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:33 smithi102 ceph-mon[29392]: pgmap v1331: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:34.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:33 smithi102 ceph-mon[37717]: pgmap v1331: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:36.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:35 smithi138 ceph-mon[33397]: pgmap v1332: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:36.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:35 smithi102 ceph-mon[29392]: pgmap v1332: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:36.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:35 smithi102 ceph-mon[37717]: pgmap v1332: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:36.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:18:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:18:36.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:18:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:18:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:18:38.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:38.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:37 smithi138 ceph-mon[33397]: pgmap v1333: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:38.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:38.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:37 smithi102 ceph-mon[29392]: pgmap v1333: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:38.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:38.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:37 smithi102 ceph-mon[37717]: pgmap v1333: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:40.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:39 smithi138 ceph-mon[33397]: pgmap v1334: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:40.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:39 smithi102 ceph-mon[29392]: pgmap v1334: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:39 smithi102 ceph-mon[37717]: pgmap v1334: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:42.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:41 smithi138 ceph-mon[33397]: pgmap v1335: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:42.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:41 smithi102 ceph-mon[29392]: pgmap v1335: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:42.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:41 smithi102 ceph-mon[37717]: pgmap v1335: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:43.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:43.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:43.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:43 smithi138 ceph-mon[33397]: pgmap v1336: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:43 smithi102 ceph-mon[29392]: pgmap v1336: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:43 smithi102 ceph-mon[37717]: pgmap v1336: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:46.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:45 smithi138 ceph-mon[33397]: pgmap v1337: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:46.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:45 smithi102 ceph-mon[29392]: pgmap v1337: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:46.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:45 smithi102 ceph-mon[37717]: pgmap v1337: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:46.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:18:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:18:46.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:18:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:18:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:18:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:47 smithi138 ceph-mon[33397]: pgmap v1338: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:47 smithi102 ceph-mon[29392]: pgmap v1338: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:47 smithi102 ceph-mon[37717]: pgmap v1338: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:50.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:49 smithi138 ceph-mon[33397]: pgmap v1339: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:50.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:49 smithi102 ceph-mon[29392]: pgmap v1339: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:50.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:49 smithi102 ceph-mon[37717]: pgmap v1339: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:52.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:51 smithi138 ceph-mon[33397]: pgmap v1340: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:52.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:51 smithi102 ceph-mon[29392]: pgmap v1340: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:52.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:51 smithi102 ceph-mon[37717]: pgmap v1340: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:54.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:53 smithi138 ceph-mon[33397]: pgmap v1341: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:54.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:53 smithi102 ceph-mon[29392]: pgmap v1341: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:54.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:53 smithi102 ceph-mon[37717]: pgmap v1341: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:56.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:55 smithi102 ceph-mon[37717]: pgmap v1342: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:56.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:55 smithi102 ceph-mon[29392]: pgmap v1342: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:56.406 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:55 smithi138 ceph-mon[33397]: pgmap v1342: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:18:56.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:18:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:18:56.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:18:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:18:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:18:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:57 smithi138 ceph-mon[33397]: pgmap v1343: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:58.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:57 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:57 smithi102 ceph-mon[29392]: pgmap v1343: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:57 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:18:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:18:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:57 smithi102 ceph-mon[37717]: pgmap v1343: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:18:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:57 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:18:59 smithi138 ceph-mon[33397]: pgmap v1344: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:00.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:18:59 smithi102 ceph-mon[29392]: pgmap v1344: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:00.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:18:59 smithi102 ceph-mon[37717]: pgmap v1344: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:02.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:01 smithi138 ceph-mon[33397]: pgmap v1345: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:02.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:01 smithi102 ceph-mon[29392]: pgmap v1345: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:02.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:01 smithi102 ceph-mon[37717]: pgmap v1345: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:04.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:03 smithi138 ceph-mon[33397]: pgmap v1346: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:04.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:03 smithi102 ceph-mon[29392]: pgmap v1346: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:04.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:03 smithi102 ceph-mon[37717]: pgmap v1346: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:06.417 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:05 smithi138 ceph-mon[33397]: pgmap v1347: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:06.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:05 smithi102 ceph-mon[29392]: pgmap v1347: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:06.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:05 smithi102 ceph-mon[37717]: pgmap v1347: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:06.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:19:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:19:06.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:19:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:19:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:19:08.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:08.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:07 smithi138 ceph-mon[33397]: pgmap v1348: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:08.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:08.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:07 smithi102 ceph-mon[37717]: pgmap v1348: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:08.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:08.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:07 smithi102 ceph-mon[29392]: pgmap v1348: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:10.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:09 smithi138 ceph-mon[33397]: pgmap v1349: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:10.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:09 smithi102 ceph-mon[29392]: pgmap v1349: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:10.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:09 smithi102 ceph-mon[37717]: pgmap v1349: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:12.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:11 smithi138 ceph-mon[33397]: pgmap v1350: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:12.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:11 smithi102 ceph-mon[29392]: pgmap v1350: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:12.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:11 smithi102 ceph-mon[37717]: pgmap v1350: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:13.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:12 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:13.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:12 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:13.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:12 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:14.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:13 smithi138 ceph-mon[33397]: pgmap v1351: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:14.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:13 smithi102 ceph-mon[29392]: pgmap v1351: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:14.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:13 smithi102 ceph-mon[37717]: pgmap v1351: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:16.428 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:15 smithi138 ceph-mon[33397]: pgmap v1352: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:16.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:15 smithi102 ceph-mon[29392]: pgmap v1352: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:16.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:15 smithi102 ceph-mon[37717]: pgmap v1352: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:16.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:19:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:19:16.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:19:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:19:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:19:18.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:18.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:17 smithi138 ceph-mon[33397]: pgmap v1353: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:18.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:18.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:17 smithi102 ceph-mon[29392]: pgmap v1353: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:18.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:18.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:17 smithi102 ceph-mon[37717]: pgmap v1353: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:20.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:19 smithi138 ceph-mon[33397]: pgmap v1354: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:20.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:19 smithi102 ceph-mon[29392]: pgmap v1354: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:20.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:19 smithi102 ceph-mon[37717]: pgmap v1354: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:22.231 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:21 smithi138 ceph-mon[33397]: pgmap v1355: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:22.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:21 smithi102 ceph-mon[29392]: pgmap v1355: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:22.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:21 smithi102 ceph-mon[37717]: pgmap v1355: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:24.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:23 smithi138 ceph-mon[33397]: pgmap v1356: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:24.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:23 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:24.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:23 smithi102 ceph-mon[37717]: pgmap v1356: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:24.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:23 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:24.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:23 smithi102 ceph-mon[29392]: pgmap v1356: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:24.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:23 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:19:25.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:25.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:24 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:25.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:24 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:19:25.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:25.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:24 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:25.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:24 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:19:25.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:19:25.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:24 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:19:25.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:24 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:19:26.334 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:25 smithi102 ceph-mon[29392]: pgmap v1357: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:26.335 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:25 smithi102 ceph-mon[37717]: pgmap v1357: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:26.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:25 smithi138 ceph-mon[33397]: pgmap v1357: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:26.683 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:19:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:19:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:19:26.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:19:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:19:28.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:28.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:27 smithi138 ceph-mon[33397]: pgmap v1358: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:28.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:27 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:27 smithi102 ceph-mon[37717]: pgmap v1358: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:27 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:27 smithi102 ceph-mon[29392]: pgmap v1358: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:27 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:30.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:29 smithi138 ceph-mon[33397]: pgmap v1359: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:30.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:29 smithi102 ceph-mon[37717]: pgmap v1359: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:30.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:29 smithi102 ceph-mon[29392]: pgmap v1359: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:32.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:31 smithi138 ceph-mon[33397]: pgmap v1360: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:32.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:31 smithi102 ceph-mon[29392]: pgmap v1360: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:32.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:31 smithi102 ceph-mon[37717]: pgmap v1360: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:34.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:34 smithi138 ceph-mon[33397]: pgmap v1361: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:34.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:34 smithi102 ceph-mon[29392]: pgmap v1361: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:34.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:34 smithi102 ceph-mon[37717]: pgmap v1361: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:36.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:36 smithi138 ceph-mon[33397]: pgmap v1362: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:36.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:36 smithi102 ceph-mon[29392]: pgmap v1362: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:36.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:36 smithi102 ceph-mon[37717]: pgmap v1362: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:36.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:19:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:19:36.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:19:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:19:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:19:38.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:38.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:38 smithi138 ceph-mon[33397]: pgmap v1363: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:38.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:38.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:38 smithi102 ceph-mon[37717]: pgmap v1363: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:38.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:38.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:38 smithi102 ceph-mon[29392]: pgmap v1363: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:40.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:40 smithi138 ceph-mon[33397]: pgmap v1364: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:40.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:40 smithi102 ceph-mon[29392]: pgmap v1364: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:40.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:40 smithi102 ceph-mon[37717]: pgmap v1364: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:42.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:42 smithi138 ceph-mon[33397]: pgmap v1365: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:42.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:42 smithi102 ceph-mon[29392]: pgmap v1365: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:42.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:42 smithi102 ceph-mon[37717]: pgmap v1365: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:43.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:43.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:43.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:44.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:44 smithi138 ceph-mon[33397]: pgmap v1366: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:44.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:44 smithi102 ceph-mon[29392]: pgmap v1366: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:44.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:44 smithi102 ceph-mon[37717]: pgmap v1366: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:46.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:46 smithi138 ceph-mon[33397]: pgmap v1367: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:46.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:46 smithi102 ceph-mon[29392]: pgmap v1367: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:46.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:46 smithi102 ceph-mon[37717]: pgmap v1367: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:46.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:19:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:19:46.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:19:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:19:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:19:48.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:48 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:48.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:48 smithi138 ceph-mon[33397]: pgmap v1368: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:48.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:48 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:48.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:48 smithi102 ceph-mon[29392]: pgmap v1368: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:48.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:48 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:48.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:48 smithi102 ceph-mon[37717]: pgmap v1368: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:50.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:50 smithi138 ceph-mon[33397]: pgmap v1369: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:50.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:50 smithi102 ceph-mon[29392]: pgmap v1369: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:50.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:50 smithi102 ceph-mon[37717]: pgmap v1369: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:52.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:52 smithi138 ceph-mon[33397]: pgmap v1370: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:52.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:52 smithi102 ceph-mon[29392]: pgmap v1370: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:52.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:52 smithi102 ceph-mon[37717]: pgmap v1370: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:54.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:54 smithi138 ceph-mon[33397]: pgmap v1371: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:54.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:54 smithi102 ceph-mon[29392]: pgmap v1371: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:54.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:54 smithi102 ceph-mon[37717]: pgmap v1371: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:56.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:56 smithi138 ceph-mon[33397]: pgmap v1372: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:56.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:56 smithi102 ceph-mon[29392]: pgmap v1372: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:56.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:56 smithi102 ceph-mon[37717]: pgmap v1372: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:19:56.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:19:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:19:56.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:19:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:19:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:19:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:58 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:58 smithi138 ceph-mon[33397]: pgmap v1373: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:19:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:58 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:58 smithi102 ceph-mon[37717]: pgmap v1373: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:19:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:58 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:58 smithi102 ceph-mon[29392]: pgmap v1373: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:19:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:19:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:00 smithi138 ceph-mon[33397]: pgmap v1374: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:00 smithi138 ceph-mon[33397]: Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-30T14:20:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:00 smithi138 ceph-mon[33397]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-30T14:20:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:00 smithi138 ceph-mon[33397]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:20:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:00 smithi138 ceph-mon[33397]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:20:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:00 smithi138 ceph-mon[33397]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[29392]: pgmap v1374: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[29392]: Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[29392]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[29392]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[29392]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[29392]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[37717]: pgmap v1374: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[37717]: Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-30T14:20:00.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[37717]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-30T14:20:00.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[37717]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T14:20:00.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[37717]: application not enabled on pool 'SetOpFlagssmithi102-76101-1' 2024-03-30T14:20:00.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:00 smithi102 ceph-mon[37717]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T14:20:02.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:02 smithi138 ceph-mon[33397]: pgmap v1375: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:02.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:02 smithi102 ceph-mon[29392]: pgmap v1375: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:02.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:02 smithi102 ceph-mon[37717]: pgmap v1375: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:04.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:04 smithi138 ceph-mon[33397]: pgmap v1376: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:04.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:04 smithi102 ceph-mon[37717]: pgmap v1376: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:04.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:04 smithi102 ceph-mon[29392]: pgmap v1376: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:06.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:06 smithi138 ceph-mon[33397]: pgmap v1377: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:06.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:06 smithi102 ceph-mon[29392]: pgmap v1377: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:06.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:06 smithi102 ceph-mon[37717]: pgmap v1377: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:06.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:20:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:20:06.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:20:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:20:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:20:08.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:08 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:08.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:08 smithi138 ceph-mon[33397]: pgmap v1378: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:08.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:08 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:08.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:08 smithi102 ceph-mon[29392]: pgmap v1378: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:08.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:08 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:08.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:08 smithi102 ceph-mon[37717]: pgmap v1378: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:10.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:10 smithi138 ceph-mon[33397]: pgmap v1379: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:10.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:10 smithi102 ceph-mon[37717]: pgmap v1379: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:10.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:10 smithi102 ceph-mon[29392]: pgmap v1379: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:12.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:12 smithi138 ceph-mon[33397]: pgmap v1380: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:12.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:12 smithi102 ceph-mon[37717]: pgmap v1380: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:12.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:12 smithi102 ceph-mon[29392]: pgmap v1380: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:13.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:13.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:13.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:14.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:14 smithi138 ceph-mon[33397]: pgmap v1381: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:14.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:14 smithi102 ceph-mon[29392]: pgmap v1381: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:14.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:14 smithi102 ceph-mon[37717]: pgmap v1381: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:16.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:16 smithi138 ceph-mon[33397]: pgmap v1382: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:16.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:16 smithi102 ceph-mon[29392]: pgmap v1382: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:16.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:16 smithi102 ceph-mon[37717]: pgmap v1382: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:16.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:20:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:20:16.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:20:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:20:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:20:18.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:18 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:18.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:18 smithi138 ceph-mon[33397]: pgmap v1383: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:18.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:18 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:18.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:18 smithi102 ceph-mon[29392]: pgmap v1383: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:18.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:18 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:18.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:18 smithi102 ceph-mon[37717]: pgmap v1383: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:20.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:20 smithi138 ceph-mon[33397]: pgmap v1384: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:20.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:20 smithi102 ceph-mon[29392]: pgmap v1384: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:20.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:20 smithi102 ceph-mon[37717]: pgmap v1384: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:22.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:22 smithi138 ceph-mon[33397]: pgmap v1385: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:22.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:22 smithi102 ceph-mon[29392]: pgmap v1385: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:22.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:22 smithi102 ceph-mon[37717]: pgmap v1385: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:24.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:24 smithi138 ceph-mon[33397]: pgmap v1386: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:24.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:24 smithi102 ceph-mon[29392]: pgmap v1386: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:24.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:24 smithi102 ceph-mon[37717]: pgmap v1386: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:25.382 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:25 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:25.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:25 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:25.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:25 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:20:26.368 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:26 smithi102 ceph-mon[37717]: pgmap v1387: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:26.368 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:26 smithi102 ceph-mon[29392]: pgmap v1387: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:26.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:26 smithi138 ceph-mon[33397]: pgmap v1387: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:26.683 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:20:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:20:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:20:26.861 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:20:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:20:28.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:28 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:28.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:28 smithi138 ceph-mon[33397]: pgmap v1388: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:28.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:28 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:28.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:28 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:28.402 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[29392]: pgmap v1388: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[37717]: pgmap v1388: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:29 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:29 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:29 smithi138 ceph-mon[33397]: pgmap v1389: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:29 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:29 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:29.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:29 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[29392]: pgmap v1389: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:29.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[37717]: pgmap v1389: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:20:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:20:29.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:29 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:20:32.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:31 smithi138 ceph-mon[33397]: pgmap v1390: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:32.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:31 smithi102 ceph-mon[29392]: pgmap v1390: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:31 smithi102 ceph-mon[37717]: pgmap v1390: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:33.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:33 smithi138 ceph-mon[33397]: pgmap v1391: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:33 smithi102 ceph-mon[29392]: pgmap v1391: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:34.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:33 smithi102 ceph-mon[37717]: pgmap v1391: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:36.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:35 smithi138 ceph-mon[33397]: pgmap v1392: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:36.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:35 smithi102 ceph-mon[29392]: pgmap v1392: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:36.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:35 smithi102 ceph-mon[37717]: pgmap v1392: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:36.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:20:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:20:36.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:20:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:20:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:20:38.117 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:38.117 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:37 smithi102 ceph-mon[37717]: pgmap v1393: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:38.117 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:38.117 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:37 smithi102 ceph-mon[29392]: pgmap v1393: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:38.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:38.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:37 smithi138 ceph-mon[33397]: pgmap v1393: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:40.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:39 smithi138 ceph-mon[33397]: pgmap v1394: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:40.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:39 smithi102 ceph-mon[29392]: pgmap v1394: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:39 smithi102 ceph-mon[37717]: pgmap v1394: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:42.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:41 smithi138 ceph-mon[33397]: pgmap v1395: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:42.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:41 smithi102 ceph-mon[29392]: pgmap v1395: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:42.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:41 smithi102 ceph-mon[37717]: pgmap v1395: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:43 smithi138 ceph-mon[33397]: pgmap v1396: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:43 smithi102 ceph-mon[29392]: pgmap v1396: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:43 smithi102 ceph-mon[37717]: pgmap v1396: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:46.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:45 smithi138 ceph-mon[33397]: pgmap v1397: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:46.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:45 smithi102 ceph-mon[29392]: pgmap v1397: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:46.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:45 smithi102 ceph-mon[37717]: pgmap v1397: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:20:46.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:20:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:20:46.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:20:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:20:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:20:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:47 smithi138 ceph-mon[33397]: pgmap v1398: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:48.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:47 smithi102 ceph-mon[29392]: pgmap v1398: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:47 smithi102 ceph-mon[37717]: pgmap v1398: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:50.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:49 smithi138 ceph-mon[33397]: pgmap v1399: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:20:50.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:49 smithi102 ceph-mon[29392]: pgmap v1399: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:20:50.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:49 smithi102 ceph-mon[37717]: pgmap v1399: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:20:52.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:51 smithi138 ceph-mon[33397]: pgmap v1400: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:52.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:51 smithi102 ceph-mon[29392]: pgmap v1400: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:52.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:51 smithi102 ceph-mon[37717]: pgmap v1400: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:54.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:53 smithi138 ceph-mon[33397]: pgmap v1401: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:20:54.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:53 smithi102 ceph-mon[29392]: pgmap v1401: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:20:54.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:53 smithi102 ceph-mon[37717]: pgmap v1401: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:20:56.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:55 smithi138 ceph-mon[33397]: pgmap v1402: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:56.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:55 smithi102 ceph-mon[29392]: pgmap v1402: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:56.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:55 smithi102 ceph-mon[37717]: pgmap v1402: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:20:56.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:20:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:20:56.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:20:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:20:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:20:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:57 smithi138 ceph-mon[33397]: pgmap v1403: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:57 smithi102 ceph-mon[29392]: pgmap v1403: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:20:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:57 smithi102 ceph-mon[37717]: pgmap v1403: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:20:59.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:59.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:20:59.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:00.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:20:59 smithi138 ceph-mon[33397]: pgmap v1404: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:21:00.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:20:59 smithi102 ceph-mon[29392]: pgmap v1404: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:21:00.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:20:59 smithi102 ceph-mon[37717]: pgmap v1404: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:21:02.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:01 smithi138 ceph-mon[33397]: pgmap v1405: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:02.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:01 smithi102 ceph-mon[29392]: pgmap v1405: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:02.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:01 smithi102 ceph-mon[37717]: pgmap v1405: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:04.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:03 smithi138 ceph-mon[33397]: pgmap v1406: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:04.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:03 smithi102 ceph-mon[29392]: pgmap v1406: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:04.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:03 smithi102 ceph-mon[37717]: pgmap v1406: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:06.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:05 smithi138 ceph-mon[33397]: pgmap v1407: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:06.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:05 smithi102 ceph-mon[29392]: pgmap v1407: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:06.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:05 smithi102 ceph-mon[37717]: pgmap v1407: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:06.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:21:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:21:06.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:21:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:21:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:21:08.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:08.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:07 smithi138 ceph-mon[33397]: pgmap v1408: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:08.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:08.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:07 smithi102 ceph-mon[37717]: pgmap v1408: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:07 smithi102 ceph-mon[29392]: pgmap v1408: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:10.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:09 smithi138 ceph-mon[33397]: pgmap v1409: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:10.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:09 smithi102 ceph-mon[29392]: pgmap v1409: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:10.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:09 smithi102 ceph-mon[37717]: pgmap v1409: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:12.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:11 smithi138 ceph-mon[33397]: pgmap v1410: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:12.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:11 smithi102 ceph-mon[29392]: pgmap v1410: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:12.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:11 smithi102 ceph-mon[37717]: pgmap v1410: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:14.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:13 smithi138 ceph-mon[33397]: pgmap v1411: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:14.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:13 smithi102 ceph-mon[29392]: pgmap v1411: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:14.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:13 smithi102 ceph-mon[37717]: pgmap v1411: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:16.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:15 smithi138 ceph-mon[33397]: pgmap v1412: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:15 smithi102 ceph-mon[37717]: pgmap v1412: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:16.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:15 smithi102 ceph-mon[29392]: pgmap v1412: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:16.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:21:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:21:16.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:21:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:21:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:21:18.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:18.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:17 smithi138 ceph-mon[33397]: pgmap v1413: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:17 smithi102 ceph-mon[29392]: pgmap v1413: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:17 smithi102 ceph-mon[37717]: pgmap v1413: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:20.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:19 smithi138 ceph-mon[33397]: pgmap v1414: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:20.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:19 smithi102 ceph-mon[29392]: pgmap v1414: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:20.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:19 smithi102 ceph-mon[37717]: pgmap v1414: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:22.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:21 smithi138 ceph-mon[33397]: pgmap v1415: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:22.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:21 smithi102 ceph-mon[29392]: pgmap v1415: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:22.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:21 smithi102 ceph-mon[37717]: pgmap v1415: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:24.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:23 smithi138 ceph-mon[33397]: pgmap v1416: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:24.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:23 smithi102 ceph-mon[29392]: pgmap v1416: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:23 smithi102 ceph-mon[37717]: pgmap v1416: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:26.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:25 smithi138 ceph-mon[33397]: pgmap v1417: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:26.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:25 smithi102 ceph-mon[29392]: pgmap v1417: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:26.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:25 smithi102 ceph-mon[37717]: pgmap v1417: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:26.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:21:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:21:26.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:21:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:21:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:21:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:27 smithi138 ceph-mon[33397]: pgmap v1418: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:28.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:27 smithi102 ceph-mon[29392]: pgmap v1418: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:28.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:28.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:27 smithi102 ceph-mon[37717]: pgmap v1418: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:29.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:29.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:29.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:30.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:29 smithi138 ceph-mon[33397]: pgmap v1419: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:30.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:29 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:21:30.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:29 smithi102 ceph-mon[29392]: pgmap v1419: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:30.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:29 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:21:30.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:29 smithi102 ceph-mon[37717]: pgmap v1419: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:30.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:29 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:21:31.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:31.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:21:31.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:30 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:21:31.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:31.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:21:31.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:30 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:21:31.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:21:31.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:21:31.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:30 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:21:32.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:31 smithi138 ceph-mon[33397]: pgmap v1420: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:32.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:31 smithi102 ceph-mon[29392]: pgmap v1420: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:32.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:31 smithi102 ceph-mon[37717]: pgmap v1420: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:34.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:33 smithi138 ceph-mon[33397]: pgmap v1421: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:34.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:33 smithi102 ceph-mon[29392]: pgmap v1421: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:34.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:33 smithi102 ceph-mon[37717]: pgmap v1421: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:36.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:35 smithi138 ceph-mon[33397]: pgmap v1422: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:36.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:35 smithi102 ceph-mon[29392]: pgmap v1422: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:36.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:35 smithi102 ceph-mon[37717]: pgmap v1422: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:36.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:21:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:21:36.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:21:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:21:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:21:38.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:38.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:37 smithi138 ceph-mon[33397]: pgmap v1423: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:38.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:38.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:37 smithi102 ceph-mon[37717]: pgmap v1423: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:38.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:38.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:37 smithi102 ceph-mon[29392]: pgmap v1423: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:40.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:39 smithi138 ceph-mon[33397]: pgmap v1424: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:40.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:39 smithi102 ceph-mon[29392]: pgmap v1424: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:40.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:39 smithi102 ceph-mon[37717]: pgmap v1424: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:42.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:41 smithi138 ceph-mon[33397]: pgmap v1425: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:42.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:41 smithi102 ceph-mon[29392]: pgmap v1425: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:42.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:41 smithi102 ceph-mon[37717]: pgmap v1425: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:43.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:43.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:43.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:44.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:43 smithi138 ceph-mon[33397]: pgmap v1426: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:44.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:43 smithi102 ceph-mon[29392]: pgmap v1426: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:44.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:43 smithi102 ceph-mon[37717]: pgmap v1426: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:46.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:45 smithi138 ceph-mon[33397]: pgmap v1427: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:46.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:45 smithi102 ceph-mon[29392]: pgmap v1427: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:46.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:45 smithi102 ceph-mon[37717]: pgmap v1427: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:46.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:21:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:21:46.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:21:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:21:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:21:48.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:48.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:47 smithi138 ceph-mon[33397]: pgmap v1428: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:48.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:48.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:47 smithi102 ceph-mon[29392]: pgmap v1428: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:48.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:48.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:47 smithi102 ceph-mon[37717]: pgmap v1428: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:50.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:49 smithi138 ceph-mon[33397]: pgmap v1429: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:21:50.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:49 smithi102 ceph-mon[29392]: pgmap v1429: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:21:50.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:49 smithi102 ceph-mon[37717]: pgmap v1429: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:21:52.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:51 smithi138 ceph-mon[33397]: pgmap v1430: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:52.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:51 smithi102 ceph-mon[29392]: pgmap v1430: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:52.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:51 smithi102 ceph-mon[37717]: pgmap v1430: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:54.245 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:53 smithi138 ceph-mon[33397]: pgmap v1431: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:54.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:53 smithi102 ceph-mon[29392]: pgmap v1431: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:54.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:53 smithi102 ceph-mon[37717]: pgmap v1431: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:56.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:56 smithi138 ceph-mon[33397]: pgmap v1432: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:56.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:56 smithi102 ceph-mon[29392]: pgmap v1432: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:56.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:56 smithi102 ceph-mon[37717]: pgmap v1432: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:21:56.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:21:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:21:56.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:21:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:21:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:21:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:58 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:58 smithi138 ceph-mon[33397]: pgmap v1433: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:58.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:21:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:58.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:58 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:58 smithi102 ceph-mon[29392]: pgmap v1433: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:58.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:21:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:21:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:58 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:21:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:58 smithi102 ceph-mon[37717]: pgmap v1433: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:21:58.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:21:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:00.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:00 smithi138 ceph-mon[33397]: pgmap v1434: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:00.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:00 smithi102 ceph-mon[29392]: pgmap v1434: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:00.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:00 smithi102 ceph-mon[37717]: pgmap v1434: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:02.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:02 smithi138 ceph-mon[33397]: pgmap v1435: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:02.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:02 smithi102 ceph-mon[29392]: pgmap v1435: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:02.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:02 smithi102 ceph-mon[37717]: pgmap v1435: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:04.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:04 smithi138 ceph-mon[33397]: pgmap v1436: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:04.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:04 smithi102 ceph-mon[29392]: pgmap v1436: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:04.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:04 smithi102 ceph-mon[37717]: pgmap v1436: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:06.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:06 smithi138 ceph-mon[33397]: pgmap v1437: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:06.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:06 smithi102 ceph-mon[29392]: pgmap v1437: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:06.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:06 smithi102 ceph-mon[37717]: pgmap v1437: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:06.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:22:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:22:06.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:22:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:22:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:22:08.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:08 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:08.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:08 smithi138 ceph-mon[33397]: pgmap v1438: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:08.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:08 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:08.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:08 smithi102 ceph-mon[29392]: pgmap v1438: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:08.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:08 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:08.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:08 smithi102 ceph-mon[37717]: pgmap v1438: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:10.428 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:10 smithi138 ceph-mon[33397]: pgmap v1439: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:10.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:10 smithi102 ceph-mon[29392]: pgmap v1439: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:10.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:10 smithi102 ceph-mon[37717]: pgmap v1439: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:12.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:12 smithi138 ceph-mon[33397]: pgmap v1440: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:12.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:12 smithi102 ceph-mon[37717]: pgmap v1440: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:12.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:12 smithi102 ceph-mon[29392]: pgmap v1440: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:13.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:13.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:13.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:14.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:14 smithi138 ceph-mon[33397]: pgmap v1441: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:14.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:14 smithi102 ceph-mon[37717]: pgmap v1441: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:14.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:14 smithi102 ceph-mon[29392]: pgmap v1441: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:16.428 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:16 smithi138 ceph-mon[33397]: pgmap v1442: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:16.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:16 smithi102 ceph-mon[29392]: pgmap v1442: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:16.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:16 smithi102 ceph-mon[37717]: pgmap v1442: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:16.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:22:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:22:16.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:22:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:22:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:22:18.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:18 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:18.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:18 smithi138 ceph-mon[33397]: pgmap v1443: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:18.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:18 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:18.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:18 smithi102 ceph-mon[29392]: pgmap v1443: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:18.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:18 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:18.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:18 smithi102 ceph-mon[37717]: pgmap v1443: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:20.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:20 smithi138 ceph-mon[33397]: pgmap v1444: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:20.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:20 smithi102 ceph-mon[29392]: pgmap v1444: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:20.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:20 smithi102 ceph-mon[37717]: pgmap v1444: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:22.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:22 smithi138 ceph-mon[33397]: pgmap v1445: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:22.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:22 smithi102 ceph-mon[29392]: pgmap v1445: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:22.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:22 smithi102 ceph-mon[37717]: pgmap v1445: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:24.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:24 smithi138 ceph-mon[33397]: pgmap v1446: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:24.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:24 smithi102 ceph-mon[29392]: pgmap v1446: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:24.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:24 smithi102 ceph-mon[37717]: pgmap v1446: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:26.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:26 smithi138 ceph-mon[33397]: pgmap v1447: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:26.432 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:26 smithi102 ceph-mon[29392]: pgmap v1447: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:26.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:26 smithi102 ceph-mon[37717]: pgmap v1447: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:26.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:22:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:22:26.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:22:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:22:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:22:28.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:28 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:28.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:28 smithi138 ceph-mon[33397]: pgmap v1448: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:28.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:28.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:28 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:28.432 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:28 smithi102 ceph-mon[37717]: pgmap v1448: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:28.433 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:28 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:28 smithi102 ceph-mon[29392]: pgmap v1448: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:28.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:30.428 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:30 smithi138 ceph-mon[33397]: pgmap v1449: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:30.429 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:30 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:30.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:30 smithi102 ceph-mon[29392]: pgmap v1449: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:30.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:30 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:30.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:30 smithi102 ceph-mon[37717]: pgmap v1449: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:30.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:30 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:22:31.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:31 smithi138 ceph-mon[33397]: pgmap v1450: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:31.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:31 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:31.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:31 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:31.929 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[29392]: pgmap v1450: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[37717]: pgmap v1450: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:22:31.933 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:22:34.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:33 smithi138 ceph-mon[33397]: pgmap v1451: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:34.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:33 smithi102 ceph-mon[29392]: pgmap v1451: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:34.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:33 smithi102 ceph-mon[37717]: pgmap v1451: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:36.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:35 smithi138 ceph-mon[33397]: pgmap v1452: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:36.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:35 smithi102 ceph-mon[29392]: pgmap v1452: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:36.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:35 smithi102 ceph-mon[37717]: pgmap v1452: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:36.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:22:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:22:36.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:22:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:22:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:22:38.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:38.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:37 smithi138 ceph-mon[33397]: pgmap v1453: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:38.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:38.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:37 smithi102 ceph-mon[29392]: pgmap v1453: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:38.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:38.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:37 smithi102 ceph-mon[37717]: pgmap v1453: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:40.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:39 smithi138 ceph-mon[33397]: pgmap v1454: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:40.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:39 smithi102 ceph-mon[29392]: pgmap v1454: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:39 smithi102 ceph-mon[37717]: pgmap v1454: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:42.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:41 smithi138 ceph-mon[33397]: pgmap v1455: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:42.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:41 smithi102 ceph-mon[37717]: pgmap v1455: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:42.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:41 smithi102 ceph-mon[29392]: pgmap v1455: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:43 smithi138 ceph-mon[33397]: pgmap v1456: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:44.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:43 smithi102 ceph-mon[37717]: pgmap v1456: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:44.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:43 smithi102 ceph-mon[29392]: pgmap v1456: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:46.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:45 smithi138 ceph-mon[33397]: pgmap v1457: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:46.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:45 smithi102 ceph-mon[37717]: pgmap v1457: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:46.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:45 smithi102 ceph-mon[29392]: pgmap v1457: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:46.928 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:22:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:22:46.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:22:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:22:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:22:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:47 smithi138 ceph-mon[33397]: pgmap v1458: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:48.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:48.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:47 smithi102 ceph-mon[29392]: pgmap v1458: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:47 smithi102 ceph-mon[37717]: pgmap v1458: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:50.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:49 smithi138 ceph-mon[33397]: pgmap v1459: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:50.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:49 smithi102 ceph-mon[29392]: pgmap v1459: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:50.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:49 smithi102 ceph-mon[37717]: pgmap v1459: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:52.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:51 smithi138 ceph-mon[33397]: pgmap v1460: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:52.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:51 smithi102 ceph-mon[29392]: pgmap v1460: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:52.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:51 smithi102 ceph-mon[37717]: pgmap v1460: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:54.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:53 smithi138 ceph-mon[33397]: pgmap v1461: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:54.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:53 smithi102 ceph-mon[37717]: pgmap v1461: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:54.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:53 smithi102 ceph-mon[29392]: pgmap v1461: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:56.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:55 smithi138 ceph-mon[33397]: pgmap v1462: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:56.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:55 smithi102 ceph-mon[37717]: pgmap v1462: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:56.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:55 smithi102 ceph-mon[29392]: pgmap v1462: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:22:56.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:22:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:22:56.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:22:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:22:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:22:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:57 smithi138 ceph-mon[33397]: pgmap v1463: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:58.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:57 smithi102 ceph-mon[29392]: pgmap v1463: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:22:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:57 smithi102 ceph-mon[37717]: pgmap v1463: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:22:59.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:59.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:22:59.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:00.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:22:59 smithi138 ceph-mon[33397]: pgmap v1464: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:00.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:22:59 smithi102 ceph-mon[29392]: pgmap v1464: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:00.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:22:59 smithi102 ceph-mon[37717]: pgmap v1464: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:02.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:01 smithi138 ceph-mon[33397]: pgmap v1465: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:02.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:01 smithi102 ceph-mon[29392]: pgmap v1465: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:02.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:01 smithi102 ceph-mon[37717]: pgmap v1465: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:04.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:03 smithi138 ceph-mon[33397]: pgmap v1466: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:04.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:03 smithi102 ceph-mon[29392]: pgmap v1466: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:04.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:03 smithi102 ceph-mon[37717]: pgmap v1466: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:06.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:05 smithi138 ceph-mon[33397]: pgmap v1467: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:06.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:05 smithi102 ceph-mon[29392]: pgmap v1467: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:06.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:05 smithi102 ceph-mon[37717]: pgmap v1467: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:06.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:23:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:23:06.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:23:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:23:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:23:08.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:08.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:07 smithi138 ceph-mon[33397]: pgmap v1468: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:08.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:08.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:07 smithi102 ceph-mon[29392]: pgmap v1468: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:08.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:08.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:07 smithi102 ceph-mon[37717]: pgmap v1468: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:10.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:09 smithi138 ceph-mon[33397]: pgmap v1469: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:10.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:09 smithi102 ceph-mon[29392]: pgmap v1469: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:10.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:09 smithi102 ceph-mon[37717]: pgmap v1469: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:12.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:11 smithi138 ceph-mon[33397]: pgmap v1470: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:12.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:11 smithi102 ceph-mon[29392]: pgmap v1470: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:12.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:11 smithi102 ceph-mon[37717]: pgmap v1470: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:14.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:13 smithi138 ceph-mon[33397]: pgmap v1471: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:14.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:14.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:13 smithi102 ceph-mon[29392]: pgmap v1471: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:14.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:13 smithi102 ceph-mon[37717]: pgmap v1471: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:16.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:15 smithi138 ceph-mon[33397]: pgmap v1472: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:16.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:15 smithi102 ceph-mon[29392]: pgmap v1472: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:16.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:15 smithi102 ceph-mon[37717]: pgmap v1472: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:16.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:23:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:23:16.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:23:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:23:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:23:18.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:18.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:17 smithi138 ceph-mon[33397]: pgmap v1473: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:18.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:17 smithi102 ceph-mon[29392]: pgmap v1473: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:17 smithi102 ceph-mon[37717]: pgmap v1473: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:20.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:19 smithi138 ceph-mon[33397]: pgmap v1474: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:20.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:19 smithi102 ceph-mon[29392]: pgmap v1474: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:20.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:19 smithi102 ceph-mon[37717]: pgmap v1474: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:22.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:21 smithi138 ceph-mon[33397]: pgmap v1475: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:22.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:21 smithi102 ceph-mon[29392]: pgmap v1475: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:22.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:21 smithi102 ceph-mon[37717]: pgmap v1475: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:24.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:23 smithi138 ceph-mon[33397]: pgmap v1476: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:24.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:23 smithi102 ceph-mon[29392]: pgmap v1476: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:24.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:23 smithi102 ceph-mon[37717]: pgmap v1476: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:26.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:25 smithi138 ceph-mon[33397]: pgmap v1477: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:26.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:25 smithi102 ceph-mon[29392]: pgmap v1477: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:26.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:25 smithi102 ceph-mon[37717]: pgmap v1477: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:26.928 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:23:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:23:26.932 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:23:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:23:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:23:28.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:27 smithi138 ceph-mon[33397]: pgmap v1478: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:28.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:28.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:27 smithi102 ceph-mon[29392]: pgmap v1478: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:28.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:28.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:27 smithi102 ceph-mon[37717]: pgmap v1478: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:29.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:29.182 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:29.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:30.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:29 smithi138 ceph-mon[33397]: pgmap v1479: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:30.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:29 smithi102 ceph-mon[29392]: pgmap v1479: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:30.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:29 smithi102 ceph-mon[37717]: pgmap v1479: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:32.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:31 smithi138 ceph-mon[33397]: pgmap v1480: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:32.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:31 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:32.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:32.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:31 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:32.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[29392]: pgmap v1480: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:32.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:32.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:32.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[37717]: pgmap v1480: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:23:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:31 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:33.178 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:32 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:33.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:32 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:33.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:32 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:33.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:32 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:33.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:32 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:33.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:32 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:33.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:32 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:23:33.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:32 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:23:33.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:32 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:23:34.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:34 smithi138 ceph-mon[33397]: pgmap v1481: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:34.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:34 smithi102 ceph-mon[29392]: pgmap v1481: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:34.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:34 smithi102 ceph-mon[37717]: pgmap v1481: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:36.676 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:36 smithi138 ceph-mon[33397]: pgmap v1482: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:36.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:36 smithi102 ceph-mon[37717]: pgmap v1482: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:36.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:36 smithi102 ceph-mon[29392]: pgmap v1482: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:36.683 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:23:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:23:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:23:36.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:23:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:23:38.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:38.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:38 smithi138 ceph-mon[33397]: pgmap v1483: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:38.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:38.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:38 smithi102 ceph-mon[29392]: pgmap v1483: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:38.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:38.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:38 smithi102 ceph-mon[37717]: pgmap v1483: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:40.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:40 smithi138 ceph-mon[33397]: pgmap v1484: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:40.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:40 smithi102 ceph-mon[29392]: pgmap v1484: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:40.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:40 smithi102 ceph-mon[37717]: pgmap v1484: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:42.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:42 smithi138 ceph-mon[33397]: pgmap v1485: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:42.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:42 smithi102 ceph-mon[29392]: pgmap v1485: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:42.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:42 smithi102 ceph-mon[37717]: pgmap v1485: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:43.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:43.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:43.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:44.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:44 smithi138 ceph-mon[33397]: pgmap v1486: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:44.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:44 smithi102 ceph-mon[29392]: pgmap v1486: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:44.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:44 smithi102 ceph-mon[37717]: pgmap v1486: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:46.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:46 smithi138 ceph-mon[33397]: pgmap v1487: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:46.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:46 smithi102 ceph-mon[37717]: pgmap v1487: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:46.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:46 smithi102 ceph-mon[29392]: pgmap v1487: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:46.683 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:23:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:23:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:23:47.178 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:23:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:23:48.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:48 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:48.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:48 smithi138 ceph-mon[33397]: pgmap v1488: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:48.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:48 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:48.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:48 smithi102 ceph-mon[37717]: pgmap v1488: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:48.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:48 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:48.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:48 smithi102 ceph-mon[29392]: pgmap v1488: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:50.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:50 smithi138 ceph-mon[33397]: pgmap v1489: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:50.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:50 smithi102 ceph-mon[29392]: pgmap v1489: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:50.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:50 smithi102 ceph-mon[37717]: pgmap v1489: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:52.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:52 smithi138 ceph-mon[33397]: pgmap v1490: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:52.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:52 smithi102 ceph-mon[29392]: pgmap v1490: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:52.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:52 smithi102 ceph-mon[37717]: pgmap v1490: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:54.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:54 smithi138 ceph-mon[33397]: pgmap v1491: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:54.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:54 smithi102 ceph-mon[29392]: pgmap v1491: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:54.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:54 smithi102 ceph-mon[37717]: pgmap v1491: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:56.678 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:56 smithi138 ceph-mon[33397]: pgmap v1492: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:56.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:56 smithi102 ceph-mon[29392]: pgmap v1492: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:56.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:56 smithi102 ceph-mon[37717]: pgmap v1492: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:23:56.683 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:23:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:23:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:23:57.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:23:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:23:58.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:58 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:58.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:58 smithi138 ceph-mon[33397]: pgmap v1493: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:58.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:23:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:58.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:58 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:58.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:58 smithi102 ceph-mon[29392]: pgmap v1493: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:58.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:23:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:23:58.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:58 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:23:58.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:58 smithi102 ceph-mon[37717]: pgmap v1493: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:23:58.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:23:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:00.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:00 smithi138 ceph-mon[33397]: pgmap v1494: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:00.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:00 smithi102 ceph-mon[29392]: pgmap v1494: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:00.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:00 smithi102 ceph-mon[37717]: pgmap v1494: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:02.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:02 smithi138 ceph-mon[33397]: pgmap v1495: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:02.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:02 smithi102 ceph-mon[29392]: pgmap v1495: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:02.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:02 smithi102 ceph-mon[37717]: pgmap v1495: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:04.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:04 smithi138 ceph-mon[33397]: pgmap v1496: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:04.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:04 smithi102 ceph-mon[29392]: pgmap v1496: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:04.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:04 smithi102 ceph-mon[37717]: pgmap v1496: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:06.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:06 smithi138 ceph-mon[33397]: pgmap v1497: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:06.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:06 smithi102 ceph-mon[37717]: pgmap v1497: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:06.683 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:06 smithi102 ceph-mon[29392]: pgmap v1497: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:06.683 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:24:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:24:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:24:07.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:24:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:24:08.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:08 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:08.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:08 smithi138 ceph-mon[33397]: pgmap v1498: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:08.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:08 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:08.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:08 smithi102 ceph-mon[29392]: pgmap v1498: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:08.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:08 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:08.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:08 smithi102 ceph-mon[37717]: pgmap v1498: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:10.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:10 smithi138 ceph-mon[33397]: pgmap v1499: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:10.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:10 smithi102 ceph-mon[29392]: pgmap v1499: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:10.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:10 smithi102 ceph-mon[37717]: pgmap v1499: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:11.679 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:11 smithi138 ceph-mon[33397]: pgmap v1500: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:11.682 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:11 smithi102 ceph-mon[29392]: pgmap v1500: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:11.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:11 smithi102 ceph-mon[37717]: pgmap v1500: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:14.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:13 smithi138 ceph-mon[33397]: pgmap v1501: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:24:14.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:14.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:13 smithi102 ceph-mon[29392]: pgmap v1501: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:24:14.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:13 smithi102 ceph-mon[37717]: pgmap v1501: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:24:14.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:16.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:15 smithi138 ceph-mon[33397]: pgmap v1502: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:16.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:15 smithi102 ceph-mon[29392]: pgmap v1502: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:16.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:15 smithi102 ceph-mon[37717]: pgmap v1502: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:16.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:24:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:24:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:24:17.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:24:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:24:18.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:18.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:17 smithi138 ceph-mon[33397]: pgmap v1503: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:18.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:18.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:17 smithi102 ceph-mon[29392]: pgmap v1503: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:18.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:17 smithi102 ceph-mon[37717]: pgmap v1503: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:20.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:19 smithi138 ceph-mon[33397]: pgmap v1504: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:20.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:19 smithi102 ceph-mon[29392]: pgmap v1504: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:20.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:19 smithi102 ceph-mon[37717]: pgmap v1504: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:22.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:21 smithi138 ceph-mon[33397]: pgmap v1505: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:22.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:21 smithi102 ceph-mon[29392]: pgmap v1505: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:22.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:21 smithi102 ceph-mon[37717]: pgmap v1505: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:24.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:23 smithi138 ceph-mon[33397]: pgmap v1506: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:24.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:23 smithi102 ceph-mon[37717]: pgmap v1506: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:24.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:23 smithi102 ceph-mon[29392]: pgmap v1506: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:26.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:25 smithi138 ceph-mon[33397]: pgmap v1507: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:26.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:25 smithi102 ceph-mon[29392]: pgmap v1507: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:26.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:25 smithi102 ceph-mon[37717]: pgmap v1507: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:26.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:24:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:24:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:24:27.080 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:24:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:24:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:28.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:27 smithi138 ceph-mon[33397]: pgmap v1508: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:28.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:27 smithi102 ceph-mon[29392]: pgmap v1508: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:28.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:28.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:27 smithi102 ceph-mon[37717]: pgmap v1508: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:29.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:29.182 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:29.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:30.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:29 smithi138 ceph-mon[33397]: pgmap v1509: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:30.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:29 smithi102 ceph-mon[37717]: pgmap v1509: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:30.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:29 smithi102 ceph-mon[29392]: pgmap v1509: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:32.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:31 smithi138 ceph-mon[33397]: pgmap v1510: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:32.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:31 smithi102 ceph-mon[37717]: pgmap v1510: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:32.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:31 smithi102 ceph-mon[29392]: pgmap v1510: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:33.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:32 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:24:33.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:32 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:24:33.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:32 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:24:34.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:33 smithi138 ceph-mon[33397]: pgmap v1511: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:34.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:33 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:24:34.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:33 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:24:34.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:33 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[29392]: pgmap v1511: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[37717]: pgmap v1511: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:24:34.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:33 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:24:36.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:35 smithi138 ceph-mon[33397]: pgmap v1512: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:36.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:35 smithi102 ceph-mon[29392]: pgmap v1512: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:36.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:35 smithi102 ceph-mon[37717]: pgmap v1512: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:36.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:24:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:24:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:24:37.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:24:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:24:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:37 smithi138 ceph-mon[33397]: pgmap v1513: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:38.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:37 smithi102 ceph-mon[29392]: pgmap v1513: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:38.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:38.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:37 smithi102 ceph-mon[37717]: pgmap v1513: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:40.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:39 smithi138 ceph-mon[33397]: pgmap v1514: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:40.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:39 smithi102 ceph-mon[29392]: pgmap v1514: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:40.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:39 smithi102 ceph-mon[37717]: pgmap v1514: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:42.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:41 smithi138 ceph-mon[33397]: pgmap v1515: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:42.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:41 smithi102 ceph-mon[29392]: pgmap v1515: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:42.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:41 smithi102 ceph-mon[37717]: pgmap v1515: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:43 smithi138 ceph-mon[33397]: pgmap v1516: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:44.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:43 smithi102 ceph-mon[29392]: pgmap v1516: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:44.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:43 smithi102 ceph-mon[37717]: pgmap v1516: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:44.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:46.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:45 smithi138 ceph-mon[33397]: pgmap v1517: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:46.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:45 smithi102 ceph-mon[29392]: pgmap v1517: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:46.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:45 smithi102 ceph-mon[37717]: pgmap v1517: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:46.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:24:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:24:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:24:47.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:24:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:24:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:47 smithi138 ceph-mon[33397]: pgmap v1518: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:48.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:47 smithi102 ceph-mon[29392]: pgmap v1518: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:48.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:47 smithi102 ceph-mon[37717]: pgmap v1518: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:48.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:50.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:49 smithi138 ceph-mon[33397]: pgmap v1519: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:50.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:49 smithi102 ceph-mon[29392]: pgmap v1519: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:50.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:49 smithi102 ceph-mon[37717]: pgmap v1519: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:52.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:51 smithi138 ceph-mon[33397]: pgmap v1520: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:52.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:51 smithi102 ceph-mon[29392]: pgmap v1520: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:52.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:51 smithi102 ceph-mon[37717]: pgmap v1520: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:54.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:53 smithi138 ceph-mon[33397]: pgmap v1521: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:54.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:53 smithi102 ceph-mon[29392]: pgmap v1521: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:54.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:53 smithi102 ceph-mon[37717]: pgmap v1521: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:56.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:55 smithi138 ceph-mon[33397]: pgmap v1522: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:56.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:55 smithi102 ceph-mon[29392]: pgmap v1522: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:56.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:55 smithi102 ceph-mon[37717]: pgmap v1522: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:24:56.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:24:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:24:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:24:57.081 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:24:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:24:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:57 smithi138 ceph-mon[33397]: pgmap v1523: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:58.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:57 smithi102 ceph-mon[37717]: pgmap v1523: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:58.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:58.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:57 smithi102 ceph-mon[29392]: pgmap v1523: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:24:58.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:24:59.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:59.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:24:59.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:00.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:24:59 smithi138 ceph-mon[33397]: pgmap v1524: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:00.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:24:59 smithi102 ceph-mon[29392]: pgmap v1524: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:00.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:24:59 smithi102 ceph-mon[37717]: pgmap v1524: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:02.179 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:01 smithi138 ceph-mon[33397]: pgmap v1525: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:02.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:01 smithi102 ceph-mon[29392]: pgmap v1525: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:02.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:01 smithi102 ceph-mon[37717]: pgmap v1525: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:04.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:03 smithi138 ceph-mon[33397]: pgmap v1526: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:04.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:03 smithi102 ceph-mon[29392]: pgmap v1526: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:04.183 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:03 smithi102 ceph-mon[37717]: pgmap v1526: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:06.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:05 smithi138 ceph-mon[33397]: pgmap v1527: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:06.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:05 smithi102 ceph-mon[29392]: pgmap v1527: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:06.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:05 smithi102 ceph-mon[37717]: pgmap v1527: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:06.933 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:25:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:25:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:25:07.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:25:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:25:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:07 smithi138 ceph-mon[33397]: pgmap v1528: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:08.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:07 smithi102 ceph-mon[29392]: pgmap v1528: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:08.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:07 smithi102 ceph-mon[37717]: pgmap v1528: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:08.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:10.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:09 smithi138 ceph-mon[33397]: pgmap v1529: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:10.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:09 smithi102 ceph-mon[29392]: pgmap v1529: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:10.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:09 smithi102 ceph-mon[37717]: pgmap v1529: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:12.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:11 smithi138 ceph-mon[33397]: pgmap v1530: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:12.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:11 smithi102 ceph-mon[29392]: pgmap v1530: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:12.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:11 smithi102 ceph-mon[37717]: pgmap v1530: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:13 smithi138 ceph-mon[33397]: pgmap v1531: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:14.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:14.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:13 smithi102 ceph-mon[29392]: pgmap v1531: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:14.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:13 smithi102 ceph-mon[37717]: pgmap v1531: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:14.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:16.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:15 smithi138 ceph-mon[33397]: pgmap v1532: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:16.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:15 smithi102 ceph-mon[29392]: pgmap v1532: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:16.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:15 smithi102 ceph-mon[37717]: pgmap v1532: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:16.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:25:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:25:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:25:17.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:25:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:25:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:17 smithi138 ceph-mon[33397]: pgmap v1533: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:18.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:17 smithi102 ceph-mon[29392]: pgmap v1533: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:17 smithi102 ceph-mon[37717]: pgmap v1533: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:18.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:20.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:19 smithi138 ceph-mon[33397]: pgmap v1534: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:20.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:19 smithi102 ceph-mon[29392]: pgmap v1534: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:20.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:19 smithi102 ceph-mon[37717]: pgmap v1534: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:22.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:21 smithi138 ceph-mon[33397]: pgmap v1535: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:22.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:21 smithi102 ceph-mon[29392]: pgmap v1535: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:22.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:21 smithi102 ceph-mon[37717]: pgmap v1535: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:24.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:23 smithi138 ceph-mon[33397]: pgmap v1536: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:24.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:23 smithi102 ceph-mon[29392]: pgmap v1536: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:24.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:23 smithi102 ceph-mon[37717]: pgmap v1536: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:26.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:25 smithi138 ceph-mon[33397]: pgmap v1537: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:26.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:25 smithi102 ceph-mon[29392]: pgmap v1537: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:26.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:25 smithi102 ceph-mon[37717]: pgmap v1537: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:26.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:25:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:25:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:25:27.081 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:25:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:25:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:27 smithi102 ceph-mon[29392]: pgmap v1538: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:28.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:28.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:27 smithi102 ceph-mon[37717]: pgmap v1538: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:28.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:28.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:27 smithi138 ceph-mon[33397]: pgmap v1538: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:28.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:29.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:29.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:29.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:30.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:29 smithi138 ceph-mon[33397]: pgmap v1539: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:30.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:29 smithi102 ceph-mon[29392]: pgmap v1539: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:30.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:29 smithi102 ceph-mon[37717]: pgmap v1539: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:32.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:31 smithi138 ceph-mon[33397]: pgmap v1540: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:32.434 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:31 smithi102 ceph-mon[29392]: pgmap v1540: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:32.434 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:31 smithi102 ceph-mon[37717]: pgmap v1540: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:34.332 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:33 smithi102 ceph-mon[29392]: pgmap v1541: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:34.332 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:33 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:25:34.332 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:33 smithi102 ceph-mon[37717]: pgmap v1541: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:34.332 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:33 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:25:34.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:33 smithi138 ceph-mon[33397]: pgmap v1541: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:34.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:33 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:25:36.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:35 smithi102 ceph-mon[29392]: pgmap v1542: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:36.080 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:35 smithi102 ceph-mon[37717]: pgmap v1542: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:36.430 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:35 smithi138 ceph-mon[33397]: pgmap v1542: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:36.763 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:25:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:25:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:25:37.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:25:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:25:38.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:38 smithi138 ceph-mon[33397]: pgmap v1543: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:38 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:38 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:38 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:25:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:38 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:25:38.681 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:38 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[29392]: pgmap v1543: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:38.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:38.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:25:38.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:25:38.685 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[37717]: pgmap v1543: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:38.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:38.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:38.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:25:38.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:25:38.686 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:38 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:25:40.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:40 smithi138 ceph-mon[33397]: pgmap v1544: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:40.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:40 smithi102 ceph-mon[29392]: pgmap v1544: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:40.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:40 smithi102 ceph-mon[37717]: pgmap v1544: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:42.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:42 smithi138 ceph-mon[33397]: pgmap v1545: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:42.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:42 smithi102 ceph-mon[29392]: pgmap v1545: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:42.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:42 smithi102 ceph-mon[37717]: pgmap v1545: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:43.680 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:43.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:43.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:44.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:44 smithi102 ceph-mon[29392]: pgmap v1546: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:44.685 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:44 smithi102 ceph-mon[37717]: pgmap v1546: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:44.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:44 smithi138 ceph-mon[33397]: pgmap v1546: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:45.930 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:45 smithi138 ceph-mon[33397]: pgmap v1547: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:45.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:45 smithi102 ceph-mon[29392]: pgmap v1547: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:45.934 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:45 smithi102 ceph-mon[37717]: pgmap v1547: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:46.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:25:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:25:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:25:47.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:25:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:25:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:47 smithi138 ceph-mon[33397]: pgmap v1548: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:48.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:47 smithi102 ceph-mon[29392]: pgmap v1548: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:48.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:48.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:47 smithi102 ceph-mon[37717]: pgmap v1548: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:48.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:50.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:49 smithi138 ceph-mon[33397]: pgmap v1549: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:50.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:49 smithi102 ceph-mon[37717]: pgmap v1549: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:50.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:49 smithi102 ceph-mon[29392]: pgmap v1549: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:52.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:51 smithi138 ceph-mon[33397]: pgmap v1550: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:52.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:51 smithi102 ceph-mon[29392]: pgmap v1550: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:52.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:51 smithi102 ceph-mon[37717]: pgmap v1550: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:54.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:53 smithi138 ceph-mon[33397]: pgmap v1551: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:54.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:53 smithi102 ceph-mon[29392]: pgmap v1551: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:54.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:53 smithi102 ceph-mon[37717]: pgmap v1551: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:56.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:55 smithi138 ceph-mon[33397]: pgmap v1552: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:56.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:55 smithi102 ceph-mon[29392]: pgmap v1552: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:56.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:55 smithi102 ceph-mon[37717]: pgmap v1552: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:25:56.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:25:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:25:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:25:57.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:25:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:25:58.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:57 smithi102 ceph-mon[29392]: pgmap v1553: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:58.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:58.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:57 smithi102 ceph-mon[37717]: pgmap v1553: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:58.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:58.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:57 smithi138 ceph-mon[33397]: pgmap v1553: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:25:58.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:25:59.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:59.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:25:59.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:00.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:25:59 smithi138 ceph-mon[33397]: pgmap v1554: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:00.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:25:59 smithi102 ceph-mon[29392]: pgmap v1554: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:00.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:25:59 smithi102 ceph-mon[37717]: pgmap v1554: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:02.180 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:01 smithi138 ceph-mon[33397]: pgmap v1555: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:02.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:01 smithi102 ceph-mon[29392]: pgmap v1555: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:02.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:01 smithi102 ceph-mon[37717]: pgmap v1555: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:03 smithi138 ceph-mon[33397]: pgmap v1556: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:04.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:03 smithi102 ceph-mon[29392]: pgmap v1556: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:04.184 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:03 smithi102 ceph-mon[37717]: pgmap v1556: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:06.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:05 smithi138 ceph-mon[33397]: pgmap v1557: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:06.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:05 smithi102 ceph-mon[29392]: pgmap v1557: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:06.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:05 smithi102 ceph-mon[37717]: pgmap v1557: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:06.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:26:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:26:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:26:07.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:26:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:26:07.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:07 smithi102 ceph-mon[29392]: pgmap v1558: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:07.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:07.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:07 smithi102 ceph-mon[37717]: pgmap v1558: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:07.935 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:08.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:07 smithi138 ceph-mon[33397]: pgmap v1558: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:08.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:10.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:09 smithi138 ceph-mon[33397]: pgmap v1559: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:10.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:09 smithi102 ceph-mon[29392]: pgmap v1559: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:10.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:09 smithi102 ceph-mon[37717]: pgmap v1559: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:11 smithi138 ceph-mon[33397]: pgmap v1560: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:12.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:11 smithi102 ceph-mon[29392]: pgmap v1560: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:12.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:11 smithi102 ceph-mon[37717]: pgmap v1560: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:13 smithi138 ceph-mon[33397]: pgmap v1561: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:13 smithi102 ceph-mon[29392]: pgmap v1561: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:14.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:13 smithi102 ceph-mon[37717]: pgmap v1561: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:14.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:15 smithi138 ceph-mon[33397]: pgmap v1562: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:16.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:15 smithi102 ceph-mon[29392]: pgmap v1562: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:16.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:15 smithi102 ceph-mon[37717]: pgmap v1562: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:16.934 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:26:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:26:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:26:17.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:26:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:26:18.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:17 smithi138 ceph-mon[33397]: pgmap v1563: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:18.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:18.184 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:17 smithi102 ceph-mon[29392]: pgmap v1563: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:18.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:18.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:17 smithi102 ceph-mon[37717]: pgmap v1563: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:18.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:20.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:19 smithi138 ceph-mon[33397]: pgmap v1564: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:20.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:19 smithi102 ceph-mon[29392]: pgmap v1564: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:20.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:19 smithi102 ceph-mon[37717]: pgmap v1564: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:22.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:21 smithi138 ceph-mon[33397]: pgmap v1565: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:22.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:21 smithi102 ceph-mon[29392]: pgmap v1565: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:22.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:21 smithi102 ceph-mon[37717]: pgmap v1565: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:24.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:23 smithi138 ceph-mon[33397]: pgmap v1566: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:24.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:23 smithi102 ceph-mon[29392]: pgmap v1566: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:24.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:23 smithi102 ceph-mon[37717]: pgmap v1566: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:26.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:25 smithi138 ceph-mon[33397]: pgmap v1567: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:26.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:25 smithi102 ceph-mon[29392]: pgmap v1567: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:26.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:25 smithi102 ceph-mon[37717]: pgmap v1567: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:26.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:26:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:26:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:26:27.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:26:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:26:28.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:27 smithi138 ceph-mon[33397]: pgmap v1568: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:28.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:28.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:27 smithi102 ceph-mon[29392]: pgmap v1568: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:28.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:28.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:27 smithi102 ceph-mon[37717]: pgmap v1568: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:28.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:29.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:29.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:29.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:30.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:29 smithi138 ceph-mon[33397]: pgmap v1569: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:30.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:29 smithi102 ceph-mon[29392]: pgmap v1569: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:30.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:29 smithi102 ceph-mon[37717]: pgmap v1569: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:32.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:31 smithi138 ceph-mon[33397]: pgmap v1570: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:32.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:31 smithi102 ceph-mon[29392]: pgmap v1570: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:32.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:31 smithi102 ceph-mon[37717]: pgmap v1570: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:34.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:33 smithi138 ceph-mon[33397]: pgmap v1571: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:34.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:33 smithi102 ceph-mon[29392]: pgmap v1571: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:34.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:33 smithi102 ceph-mon[37717]: pgmap v1571: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:36.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:35 smithi138 ceph-mon[33397]: pgmap v1572: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:36.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:35 smithi102 ceph-mon[29392]: pgmap v1572: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:36.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:35 smithi102 ceph-mon[37717]: pgmap v1572: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:36.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:26:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:26:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:26:37.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:26:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:37 smithi138 ceph-mon[33397]: pgmap v1573: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:37 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:26:38.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:37 smithi102 ceph-mon[29392]: pgmap v1573: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:38.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:38.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:37 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:26:38.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:37 smithi102 ceph-mon[37717]: pgmap v1573: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:38.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:38.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:37 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:26:40.086 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:39 smithi138 ceph-mon[33397]: pgmap v1574: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:40.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:39 smithi102 ceph-mon[29392]: pgmap v1574: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:40.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:39 smithi102 ceph-mon[37717]: pgmap v1574: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:42.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:41 smithi138 ceph-mon[33397]: pgmap v1575: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:42.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.181 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:41 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:26:42.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:41 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:26:42.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:41 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[29392]: pgmap v1575: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:26:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:26:42.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[37717]: pgmap v1575: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:42.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:42.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:26:42.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:26:42.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:41 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:26:44.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:43 smithi102 ceph-mon[37717]: pgmap v1576: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:44.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:43 smithi102 ceph-mon[29392]: pgmap v1576: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:44.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:44.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:43 smithi138 ceph-mon[33397]: pgmap v1576: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:44.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:46.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:45 smithi102 ceph-mon[29392]: pgmap v1577: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:45 smithi102 ceph-mon[37717]: pgmap v1577: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:45 smithi138 ceph-mon[33397]: pgmap v1577: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:46.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:26:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:26:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:26:47.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:26:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:26:48.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:47 smithi138 ceph-mon[33397]: pgmap v1578: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:48.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:47 smithi102 ceph-mon[37717]: pgmap v1578: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:48.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:47 smithi102 ceph-mon[29392]: pgmap v1578: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:48.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:50.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:49 smithi138 ceph-mon[33397]: pgmap v1579: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:50.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:49 smithi102 ceph-mon[29392]: pgmap v1579: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:50.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:49 smithi102 ceph-mon[37717]: pgmap v1579: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:52.258 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:51 smithi138 ceph-mon[33397]: pgmap v1580: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:52.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:51 smithi102 ceph-mon[29392]: pgmap v1580: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:52.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:51 smithi102 ceph-mon[37717]: pgmap v1580: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:26:54.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:53 smithi138 ceph-mon[33397]: pgmap v1581: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:54.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:53 smithi102 ceph-mon[37717]: pgmap v1581: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:54.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:53 smithi102 ceph-mon[29392]: pgmap v1581: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:56.238 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:55 smithi102 ceph-mon[29392]: pgmap v1582: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:56.238 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:55 smithi102 ceph-mon[37717]: pgmap v1582: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:56.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:55 smithi138 ceph-mon[33397]: pgmap v1582: 196 pgs: 196 active+clean; 582 KiB data, 769 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:26:56.935 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:26:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:26:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:26:57.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:26:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:26:58.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:57 smithi138 ceph-mon[33397]: pgmap v1583: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:26:58.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:58.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:57 smithi102 ceph-mon[29392]: pgmap v1583: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:26:58.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:58.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:57 smithi102 ceph-mon[37717]: pgmap v1583: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:26:58.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:26:59.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:59.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:26:59.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:00.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:26:59 smithi138 ceph-mon[33397]: pgmap v1584: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:00.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:26:59 smithi102 ceph-mon[29392]: pgmap v1584: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:26:59 smithi102 ceph-mon[37717]: pgmap v1584: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:02.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:01 smithi138 ceph-mon[33397]: pgmap v1585: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:02.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:01 smithi102 ceph-mon[29392]: pgmap v1585: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:02.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:01 smithi102 ceph-mon[37717]: pgmap v1585: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:04.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:03 smithi138 ceph-mon[33397]: pgmap v1586: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:04.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:03 smithi102 ceph-mon[29392]: pgmap v1586: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:04.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:03 smithi102 ceph-mon[37717]: pgmap v1586: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:06.431 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:05 smithi138 ceph-mon[33397]: pgmap v1587: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:06.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:05 smithi102 ceph-mon[29392]: pgmap v1587: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:06.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:05 smithi102 ceph-mon[37717]: pgmap v1587: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:06.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:27:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:27:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:27:07.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:27:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:27:08.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:07 smithi138 ceph-mon[33397]: pgmap v1588: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:08.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:08.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:07 smithi102 ceph-mon[29392]: pgmap v1588: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:08.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:08.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:07 smithi102 ceph-mon[37717]: pgmap v1588: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-30T14:27:08.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:10.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:09 smithi138 ceph-mon[33397]: pgmap v1589: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:10.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:09 smithi102 ceph-mon[29392]: pgmap v1589: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:10.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:09 smithi102 ceph-mon[37717]: pgmap v1589: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:12.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:11 smithi138 ceph-mon[33397]: pgmap v1590: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:12.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:11 smithi102 ceph-mon[29392]: pgmap v1590: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:12.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:11 smithi102 ceph-mon[37717]: pgmap v1590: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:14.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:13 smithi138 ceph-mon[33397]: pgmap v1591: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:14.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:14.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:13 smithi102 ceph-mon[29392]: pgmap v1591: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:14.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:14.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:13 smithi102 ceph-mon[37717]: pgmap v1591: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:14.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:16.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:15 smithi138 ceph-mon[33397]: pgmap v1592: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:16.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:15 smithi102 ceph-mon[29392]: pgmap v1592: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:16.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:15 smithi102 ceph-mon[37717]: pgmap v1592: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:16.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:27:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:27:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:27:17.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:27:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:27:18.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:17 smithi138 ceph-mon[33397]: pgmap v1593: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:18.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:18.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:17 smithi102 ceph-mon[37717]: pgmap v1593: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:18.435 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:18.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:17 smithi102 ceph-mon[29392]: pgmap v1593: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:18.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:20.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:19 smithi138 ceph-mon[33397]: pgmap v1594: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:20.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:19 smithi102 ceph-mon[37717]: pgmap v1594: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:20.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:19 smithi102 ceph-mon[29392]: pgmap v1594: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:22.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:22 smithi138 ceph-mon[33397]: pgmap v1595: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:22.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:22 smithi102 ceph-mon[29392]: pgmap v1595: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:22.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:22 smithi102 ceph-mon[37717]: pgmap v1595: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:24.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:24 smithi138 ceph-mon[33397]: pgmap v1596: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:24.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:24 smithi102 ceph-mon[37717]: pgmap v1596: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:24.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:24 smithi102 ceph-mon[29392]: pgmap v1596: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:27:26.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:26 smithi138 ceph-mon[33397]: pgmap v1597: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:26.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:26 smithi102 ceph-mon[29392]: pgmap v1597: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:26.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:26 smithi102 ceph-mon[37717]: pgmap v1597: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:26.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:27:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:27:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:27:27.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:27:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:27:28.369 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:28 smithi138 ceph-mon[33397]: pgmap v1598: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:28.369 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:28 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:28 smithi102 ceph-mon[37717]: pgmap v1598: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:28.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:28 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:28.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:28 smithi102 ceph-mon[29392]: pgmap v1598: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:28.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:28 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:29.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:29 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:29.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:29 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:29.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:29 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:30.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:30 smithi138 ceph-mon[33397]: pgmap v1599: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:30.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:30 smithi102 ceph-mon[29392]: pgmap v1599: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:30.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:30 smithi102 ceph-mon[37717]: pgmap v1599: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:32.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:32 smithi138 ceph-mon[33397]: pgmap v1600: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:32.435 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:32 smithi102 ceph-mon[29392]: pgmap v1600: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:32.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:32 smithi102 ceph-mon[37717]: pgmap v1600: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:34.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:34 smithi138 ceph-mon[33397]: pgmap v1601: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:34.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:34 smithi102 ceph-mon[29392]: pgmap v1601: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:34.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:34 smithi102 ceph-mon[37717]: pgmap v1601: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:36.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:36 smithi138 ceph-mon[33397]: pgmap v1602: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:36.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:36 smithi102 ceph-mon[29392]: pgmap v1602: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:36.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:36 smithi102 ceph-mon[37717]: pgmap v1602: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:36.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:27:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:27:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:27:37.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:27:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:27:38.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:38 smithi138 ceph-mon[33397]: pgmap v1603: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:38.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:38.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:38 smithi102 ceph-mon[37717]: pgmap v1603: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:38.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:38.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:38 smithi102 ceph-mon[29392]: pgmap v1603: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:38.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:40.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:40 smithi138 ceph-mon[33397]: pgmap v1604: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:40.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:40 smithi102 ceph-mon[29392]: pgmap v1604: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:40.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:40 smithi102 ceph-mon[37717]: pgmap v1604: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:42.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:42 smithi138 ceph-mon[33397]: pgmap v1605: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:42.432 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:42.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:42 smithi102 ceph-mon[29392]: pgmap v1605: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:42.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:42.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:42 smithi102 ceph-mon[37717]: pgmap v1605: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:42.436 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:27:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:43 smithi138 ceph-mon[33397]: pgmap v1606: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:44.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:43 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:43 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:27:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:43 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[37717]: pgmap v1606: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:27:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[29392]: pgmap v1606: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:27:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:27:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:27:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:46.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:45 smithi138 ceph-mon[33397]: pgmap v1607: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:46.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:45 smithi102 ceph-mon[29392]: pgmap v1607: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:46.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:45 smithi102 ceph-mon[37717]: pgmap v1607: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:46.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:27:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:27:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:27:47.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:27:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:27:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:47 smithi138 ceph-mon[33397]: pgmap v1608: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:48.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:47 smithi102 ceph-mon[37717]: pgmap v1608: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:48.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:48.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:47 smithi102 ceph-mon[29392]: pgmap v1608: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:48.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:50.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:49 smithi138 ceph-mon[33397]: pgmap v1609: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:50.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:49 smithi102 ceph-mon[29392]: pgmap v1609: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:50.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:49 smithi102 ceph-mon[37717]: pgmap v1609: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:52.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:51 smithi138 ceph-mon[33397]: pgmap v1610: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:52.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:51 smithi102 ceph-mon[29392]: pgmap v1610: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:52.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:51 smithi102 ceph-mon[37717]: pgmap v1610: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:54.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:53 smithi138 ceph-mon[33397]: pgmap v1611: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:54.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:53 smithi102 ceph-mon[29392]: pgmap v1611: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:54.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:53 smithi102 ceph-mon[37717]: pgmap v1611: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:56.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:55 smithi138 ceph-mon[33397]: pgmap v1612: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:56.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:55 smithi102 ceph-mon[29392]: pgmap v1612: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:56.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:55 smithi102 ceph-mon[37717]: pgmap v1612: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:27:56.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:27:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:27:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:27:57.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:27:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:27:58.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:57 smithi138 ceph-mon[33397]: pgmap v1613: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:58.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:58.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:57 smithi102 ceph-mon[29392]: pgmap v1613: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:58.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:58.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:57 smithi102 ceph-mon[37717]: pgmap v1613: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:27:58.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:27:59.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:59.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:27:59.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:00.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:27:59 smithi138 ceph-mon[33397]: pgmap v1614: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:00.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:27:59 smithi102 ceph-mon[29392]: pgmap v1614: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:00.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:27:59 smithi102 ceph-mon[37717]: pgmap v1614: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:02.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:01 smithi138 ceph-mon[33397]: pgmap v1615: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:02.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:01 smithi102 ceph-mon[29392]: pgmap v1615: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:02.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:01 smithi102 ceph-mon[37717]: pgmap v1615: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:04.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:03 smithi138 ceph-mon[33397]: pgmap v1616: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:04.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:03 smithi102 ceph-mon[29392]: pgmap v1616: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:04.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:03 smithi102 ceph-mon[37717]: pgmap v1616: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:06.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:05 smithi138 ceph-mon[33397]: pgmap v1617: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:06.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:05 smithi102 ceph-mon[29392]: pgmap v1617: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:06.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:05 smithi102 ceph-mon[37717]: pgmap v1617: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:06.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:28:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:28:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:28:07.432 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:28:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:28:08.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:07 smithi138 ceph-mon[33397]: pgmap v1618: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:08.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:08.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:07 smithi102 ceph-mon[29392]: pgmap v1618: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:08.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:08.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:07 smithi102 ceph-mon[37717]: pgmap v1618: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:08.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:10.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:09 smithi138 ceph-mon[33397]: pgmap v1619: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:10.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:09 smithi102 ceph-mon[29392]: pgmap v1619: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:10.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:09 smithi102 ceph-mon[37717]: pgmap v1619: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:12.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:11 smithi138 ceph-mon[33397]: pgmap v1620: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:12.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:11 smithi102 ceph-mon[29392]: pgmap v1620: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:12.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:11 smithi102 ceph-mon[37717]: pgmap v1620: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:14.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:13 smithi138 ceph-mon[33397]: pgmap v1621: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:14.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:13 smithi102 ceph-mon[29392]: pgmap v1621: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:14.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:13 smithi102 ceph-mon[37717]: pgmap v1621: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:14.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:16.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:15 smithi138 ceph-mon[33397]: pgmap v1622: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:16.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:15 smithi102 ceph-mon[29392]: pgmap v1622: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:16.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:15 smithi102 ceph-mon[37717]: pgmap v1622: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:16.936 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:28:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:28:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:28:17.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:28:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:28:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:17 smithi138 ceph-mon[33397]: pgmap v1623: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:18.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:17 smithi102 ceph-mon[29392]: pgmap v1623: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:18.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:18.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:17 smithi102 ceph-mon[37717]: pgmap v1623: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:18.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:20.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:19 smithi138 ceph-mon[33397]: pgmap v1624: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:20.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:19 smithi102 ceph-mon[37717]: pgmap v1624: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:20.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:19 smithi102 ceph-mon[29392]: pgmap v1624: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:21.433 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:28:21 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=cleanup t=2024-03-30T14:28:21.034450008Z level=info msg="Completed cleanup jobs" duration=4.687705ms 2024-03-30T14:28:22.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:21 smithi138 ceph-mon[33397]: pgmap v1625: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:22.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:21 smithi102 ceph-mon[29392]: pgmap v1625: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:22.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:21 smithi102 ceph-mon[37717]: pgmap v1625: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:24.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:23 smithi138 ceph-mon[33397]: pgmap v1626: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:28:24.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:23 smithi102 ceph-mon[29392]: pgmap v1626: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:28:24.190 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:23 smithi102 ceph-mon[37717]: pgmap v1626: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:28:26.182 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:25 smithi138 ceph-mon[33397]: pgmap v1627: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:26.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:25 smithi102 ceph-mon[29392]: pgmap v1627: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:26.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:25 smithi102 ceph-mon[37717]: pgmap v1627: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:26.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:28:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:28:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:28:27.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:28:26 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:28:28.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:27 smithi138 ceph-mon[33397]: pgmap v1628: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:28.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:28.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:27 smithi102 ceph-mon[29392]: pgmap v1628: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:28.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:28.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:27 smithi102 ceph-mon[37717]: pgmap v1628: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:28.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:29.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:29.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:29.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:30.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:29 smithi138 ceph-mon[33397]: pgmap v1629: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:28:30.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:29 smithi102 ceph-mon[29392]: pgmap v1629: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:28:30.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:29 smithi102 ceph-mon[37717]: pgmap v1629: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:28:32.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:31 smithi138 ceph-mon[33397]: pgmap v1630: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:32.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:31 smithi102 ceph-mon[29392]: pgmap v1630: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:32.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:31 smithi102 ceph-mon[37717]: pgmap v1630: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:34.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:33 smithi138 ceph-mon[33397]: pgmap v1631: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:34.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:33 smithi102 ceph-mon[29392]: pgmap v1631: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:34.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:33 smithi102 ceph-mon[37717]: pgmap v1631: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:36.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:35 smithi138 ceph-mon[33397]: pgmap v1632: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:36.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:35 smithi102 ceph-mon[29392]: pgmap v1632: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:36.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:35 smithi102 ceph-mon[37717]: pgmap v1632: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:36.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:28:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:28:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:28:37.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:28:36 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:28:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:37 smithi138 ceph-mon[33397]: pgmap v1633: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:38.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:37 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:38.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:37 smithi102 ceph-mon[29392]: pgmap v1633: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:38.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:37 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:37 smithi102 ceph-mon[37717]: pgmap v1633: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:37 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:40.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:39 smithi138 ceph-mon[33397]: pgmap v1634: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:40.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:39 smithi102 ceph-mon[29392]: pgmap v1634: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:40.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:39 smithi102 ceph-mon[37717]: pgmap v1634: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:42.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:41 smithi138 ceph-mon[33397]: pgmap v1635: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:42.186 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:41 smithi102 ceph-mon[29392]: pgmap v1635: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:42.187 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:41 smithi102 ceph-mon[37717]: pgmap v1635: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:43.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:42 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:43.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:42 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:43.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:42 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:28:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:43 smithi138 ceph-mon[33397]: pgmap v1636: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:44.183 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:43 smithi102 ceph-mon[37717]: pgmap v1636: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:44.186 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:43 smithi102 ceph-mon[29392]: pgmap v1636: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:44.187 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:45.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:44 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:44 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:44 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:45.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:44 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:45.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:44 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:45.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:28:45.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:28:45.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:44 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:28:46.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:46 smithi138 ceph-mon[33397]: pgmap v1637: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:46.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:46 smithi102 ceph-mon[29392]: pgmap v1637: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:46.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:46 smithi102 ceph-mon[37717]: pgmap v1637: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:46.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:28:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:28:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:28:47.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:28:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:28:48.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:48 smithi138 ceph-mon[33397]: pgmap v1638: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:48.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:48 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:48 smithi102 ceph-mon[29392]: pgmap v1638: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:48.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:48 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:48.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:48 smithi102 ceph-mon[37717]: pgmap v1638: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:48.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:48 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:50.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:50 smithi138 ceph-mon[33397]: pgmap v1639: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:50.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:50 smithi102 ceph-mon[29392]: pgmap v1639: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:50.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:50 smithi102 ceph-mon[37717]: pgmap v1639: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:52.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:52 smithi138 ceph-mon[33397]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:52.436 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:52 smithi102 ceph-mon[29392]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:52.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:52 smithi102 ceph-mon[37717]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:54.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:54 smithi138 ceph-mon[33397]: pgmap v1641: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:54.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:54 smithi102 ceph-mon[29392]: pgmap v1641: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:54.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:54 smithi102 ceph-mon[37717]: pgmap v1641: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:56.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:56 smithi138 ceph-mon[33397]: pgmap v1642: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:56.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:56 smithi102 ceph-mon[29392]: pgmap v1642: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:56.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:56 smithi102 ceph-mon[37717]: pgmap v1642: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:28:56.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:28:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:28:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:28:57.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:28:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:28:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:58 smithi138 ceph-mon[33397]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:58.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:58 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:58.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:58 smithi102 ceph-mon[37717]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:58.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:58 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:58 smithi102 ceph-mon[29392]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:28:58.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:58 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:28:59.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:28:59 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:59.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:28:59 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:28:59.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:28:59 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:00.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:00 smithi138 ceph-mon[33397]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:00.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:00 smithi102 ceph-mon[29392]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:00.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:00 smithi102 ceph-mon[37717]: pgmap v1644: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:02.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:02 smithi138 ceph-mon[33397]: pgmap v1645: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:02.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:02 smithi102 ceph-mon[37717]: pgmap v1645: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:02.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:02 smithi102 ceph-mon[29392]: pgmap v1645: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:04.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:04 smithi138 ceph-mon[33397]: pgmap v1646: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:04.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:04 smithi102 ceph-mon[29392]: pgmap v1646: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:04.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:04 smithi102 ceph-mon[37717]: pgmap v1646: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:06.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:06 smithi138 ceph-mon[33397]: pgmap v1647: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:06.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:06 smithi102 ceph-mon[37717]: pgmap v1647: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:06.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:06 smithi102 ceph-mon[29392]: pgmap v1647: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:06.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:29:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:29:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:29:07.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:29:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:29:08.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:08 smithi138 ceph-mon[33397]: pgmap v1648: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:08.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:08 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:08.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:08 smithi102 ceph-mon[37717]: pgmap v1648: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:08.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:08 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:08.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:08 smithi102 ceph-mon[29392]: pgmap v1648: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:08.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:08 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:10.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:10 smithi138 ceph-mon[33397]: pgmap v1649: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:10.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:10 smithi102 ceph-mon[29392]: pgmap v1649: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:10.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:10 smithi102 ceph-mon[37717]: pgmap v1649: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:12.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:12 smithi138 ceph-mon[33397]: pgmap v1650: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:12.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:12 smithi102 ceph-mon[29392]: pgmap v1650: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:12.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:12 smithi102 ceph-mon[37717]: pgmap v1650: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:14.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:14 smithi138 ceph-mon[33397]: pgmap v1651: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:14.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:14 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:14.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:14 smithi102 ceph-mon[37717]: pgmap v1651: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:14.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:14 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:14.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:14 smithi102 ceph-mon[29392]: pgmap v1651: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:14.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:14 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:16.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:16 smithi138 ceph-mon[33397]: pgmap v1652: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:16.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:16 smithi102 ceph-mon[29392]: pgmap v1652: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:16.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:16 smithi102 ceph-mon[37717]: pgmap v1652: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:16.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:29:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:29:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:29:17.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:29:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:29:18.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:18 smithi138 ceph-mon[33397]: pgmap v1653: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:18.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:18 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:18.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:18 smithi102 ceph-mon[29392]: pgmap v1653: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:18.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:18 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:18.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:18 smithi102 ceph-mon[37717]: pgmap v1653: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:18.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:18 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:20.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:20 smithi138 ceph-mon[33397]: pgmap v1654: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:20.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:20 smithi102 ceph-mon[29392]: pgmap v1654: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:20.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:20 smithi102 ceph-mon[37717]: pgmap v1654: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:29:22.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:22 smithi138 ceph-mon[33397]: pgmap v1655: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:22.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:22 smithi102 ceph-mon[29392]: pgmap v1655: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:22.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:22 smithi102 ceph-mon[37717]: pgmap v1655: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:24.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:24 smithi138 ceph-mon[33397]: pgmap v1656: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:24.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:24 smithi102 ceph-mon[29392]: pgmap v1656: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:24.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:24 smithi102 ceph-mon[37717]: pgmap v1656: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:29:26.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:26 smithi138 ceph-mon[33397]: pgmap v1657: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:26.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:26 smithi102 ceph-mon[37717]: pgmap v1657: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:26.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:26 smithi102 ceph-mon[29392]: pgmap v1657: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:26.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:29:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:29:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:29:27.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:29:27 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:29:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:28 smithi138 ceph-mon[33397]: pgmap v1658: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:28.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:28 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:28.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:28 smithi102 ceph-mon[29392]: pgmap v1658: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:28.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:28 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:28 smithi102 ceph-mon[37717]: pgmap v1658: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:28.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:28 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:29.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:29 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:29.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:29 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:29.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:29 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:30.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:30 smithi138 ceph-mon[33397]: pgmap v1659: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:30.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:30 smithi102 ceph-mon[29392]: pgmap v1659: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:30.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:30 smithi102 ceph-mon[37717]: pgmap v1659: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:32.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:32 smithi138 ceph-mon[33397]: pgmap v1660: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:32.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:32 smithi102 ceph-mon[29392]: pgmap v1660: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:32.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:32 smithi102 ceph-mon[37717]: pgmap v1660: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:34.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:34 smithi138 ceph-mon[33397]: pgmap v1661: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:34.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:34 smithi138 ceph-mon[33397]: osdmap e767: 8 total, 8 up, 8 in 2024-03-30T14:29:34.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:34 smithi102 ceph-mon[29392]: pgmap v1661: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:34.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:34 smithi102 ceph-mon[29392]: osdmap e767: 8 total, 8 up, 8 in 2024-03-30T14:29:34.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:34 smithi102 ceph-mon[37717]: pgmap v1661: 196 pgs: 196 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:29:34.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:34 smithi102 ceph-mon[37717]: osdmap e767: 8 total, 8 up, 8 in 2024-03-30T14:29:35.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:35 smithi138 ceph-mon[33397]: osdmap e768: 8 total, 8 up, 8 in 2024-03-30T14:29:35.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:35 smithi102 ceph-mon[29392]: osdmap e768: 8 total, 8 up, 8 in 2024-03-30T14:29:35.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:35 smithi102 ceph-mon[37717]: osdmap e768: 8 total, 8 up, 8 in 2024-03-30T14:29:36.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:36 smithi138 ceph-mon[33397]: pgmap v1664: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:29:36.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:36 smithi138 ceph-mon[33397]: osdmap e769: 8 total, 8 up, 8 in 2024-03-30T14:29:36.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:36 smithi102 ceph-mon[29392]: pgmap v1664: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:29:36.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:36 smithi102 ceph-mon[29392]: osdmap e769: 8 total, 8 up, 8 in 2024-03-30T14:29:36.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:36 smithi102 ceph-mon[37717]: pgmap v1664: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:29:36.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:36 smithi102 ceph-mon[37717]: osdmap e769: 8 total, 8 up, 8 in 2024-03-30T14:29:36.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:29:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:29:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:29:37.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:37 smithi138 ceph-mon[33397]: osdmap e770: 8 total, 8 up, 8 in 2024-03-30T14:29:37.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:29:37 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:29:37.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:37 smithi102 ceph-mon[29392]: osdmap e770: 8 total, 8 up, 8 in 2024-03-30T14:29:37.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:37 smithi102 ceph-mon[37717]: osdmap e770: 8 total, 8 up, 8 in 2024-03-30T14:29:38.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:38 smithi138 ceph-mon[33397]: pgmap v1667: 164 pgs: 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:38 smithi138 ceph-mon[33397]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:38.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:38 smithi138 ceph-mon[33397]: osdmap e771: 8 total, 8 up, 8 in 2024-03-30T14:29:38.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[37717]: pgmap v1667: 164 pgs: 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:38.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:38.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[37717]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:38.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[37717]: osdmap e771: 8 total, 8 up, 8 in 2024-03-30T14:29:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[29392]: pgmap v1667: 164 pgs: 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[29392]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:38.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:38 smithi102 ceph-mon[29392]: osdmap e771: 8 total, 8 up, 8 in 2024-03-30T14:29:39.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:39 smithi138 ceph-mon[33397]: osdmap e772: 8 total, 8 up, 8 in 2024-03-30T14:29:39.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:39 smithi102 ceph-mon[29392]: osdmap e772: 8 total, 8 up, 8 in 2024-03-30T14:29:39.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:39 smithi102 ceph-mon[37717]: osdmap e772: 8 total, 8 up, 8 in 2024-03-30T14:29:40.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:40 smithi138 ceph-mon[33397]: pgmap v1670: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:40.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:40 smithi138 ceph-mon[33397]: osdmap e773: 8 total, 8 up, 8 in 2024-03-30T14:29:40.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:40 smithi102 ceph-mon[29392]: pgmap v1670: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:40.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:40 smithi102 ceph-mon[29392]: osdmap e773: 8 total, 8 up, 8 in 2024-03-30T14:29:40.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:40 smithi102 ceph-mon[37717]: pgmap v1670: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:40.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:40 smithi102 ceph-mon[37717]: osdmap e773: 8 total, 8 up, 8 in 2024-03-30T14:29:41.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:41 smithi138 ceph-mon[33397]: osdmap e774: 8 total, 8 up, 8 in 2024-03-30T14:29:41.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:41 smithi102 ceph-mon[29392]: osdmap e774: 8 total, 8 up, 8 in 2024-03-30T14:29:41.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:41 smithi102 ceph-mon[37717]: osdmap e774: 8 total, 8 up, 8 in 2024-03-30T14:29:42.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:42 smithi138 ceph-mon[33397]: pgmap v1673: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:42.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:42 smithi138 ceph-mon[33397]: osdmap e775: 8 total, 8 up, 8 in 2024-03-30T14:29:42.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:42 smithi102 ceph-mon[29392]: pgmap v1673: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:42.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:42 smithi102 ceph-mon[29392]: osdmap e775: 8 total, 8 up, 8 in 2024-03-30T14:29:42.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:42 smithi102 ceph-mon[37717]: pgmap v1673: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:42.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:42 smithi102 ceph-mon[37717]: osdmap e775: 8 total, 8 up, 8 in 2024-03-30T14:29:43.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:43 smithi138 ceph-mon[33397]: osdmap e776: 8 total, 8 up, 8 in 2024-03-30T14:29:43.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:43 smithi102 ceph-mon[29392]: osdmap e776: 8 total, 8 up, 8 in 2024-03-30T14:29:43.437 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:43 smithi102 ceph-mon[37717]: osdmap e776: 8 total, 8 up, 8 in 2024-03-30T14:29:44.433 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:44 smithi138 ceph-mon[33397]: pgmap v1676: 164 pgs: 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:44.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:44 smithi138 ceph-mon[33397]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:44.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:44 smithi138 ceph-mon[33397]: osdmap e777: 8 total, 8 up, 8 in 2024-03-30T14:29:44.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:44 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:44.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:44 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:29:44.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[29392]: pgmap v1676: 164 pgs: 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:44.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[29392]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:44.437 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[29392]: osdmap e777: 8 total, 8 up, 8 in 2024-03-30T14:29:44.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:44.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:29:44.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[37717]: pgmap v1676: 164 pgs: 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:44.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[37717]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:44.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[37717]: osdmap e777: 8 total, 8 up, 8 in 2024-03-30T14:29:44.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:44.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:44 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:29:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:45 smithi138 ceph-mon[33397]: osdmap e778: 8 total, 8 up, 8 in 2024-03-30T14:29:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:45 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:45 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:29:45.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:45 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:29:45.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[29392]: osdmap e778: 8 total, 8 up, 8 in 2024-03-30T14:29:45.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:45.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:29:45.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:29:45.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[37717]: osdmap e778: 8 total, 8 up, 8 in 2024-03-30T14:29:45.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:29:45.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:29:45.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:45 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:29:46.508 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:46 smithi102 ceph-mon[29392]: pgmap v1679: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:29:46.508 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:46 smithi102 ceph-mon[29392]: osdmap e779: 8 total, 8 up, 8 in 2024-03-30T14:29:46.508 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:46 smithi102 ceph-mon[37717]: pgmap v1679: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:29:46.508 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:46 smithi102 ceph-mon[37717]: osdmap e779: 8 total, 8 up, 8 in 2024-03-30T14:29:46.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:46 smithi138 ceph-mon[33397]: pgmap v1679: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:29:46.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:46 smithi138 ceph-mon[33397]: osdmap e779: 8 total, 8 up, 8 in 2024-03-30T14:29:46.937 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:29:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:29:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:29:47.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:47 smithi138 ceph-mon[33397]: osdmap e780: 8 total, 8 up, 8 in 2024-03-30T14:29:47.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:29:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:29:47.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:47 smithi102 ceph-mon[29392]: osdmap e780: 8 total, 8 up, 8 in 2024-03-30T14:29:47.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:47 smithi102 ceph-mon[37717]: osdmap e780: 8 total, 8 up, 8 in 2024-03-30T14:29:48.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:48 smithi138 ceph-mon[33397]: pgmap v1682: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:48.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:48 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:48.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:48 smithi138 ceph-mon[33397]: osdmap e781: 8 total, 8 up, 8 in 2024-03-30T14:29:48.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:48 smithi102 ceph-mon[29392]: pgmap v1682: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:48.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:48 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:48.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:48 smithi102 ceph-mon[29392]: osdmap e781: 8 total, 8 up, 8 in 2024-03-30T14:29:48.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:48 smithi102 ceph-mon[37717]: pgmap v1682: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:48.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:48 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:48.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:48 smithi102 ceph-mon[37717]: osdmap e781: 8 total, 8 up, 8 in 2024-03-30T14:29:49.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:49 smithi138 ceph-mon[33397]: osdmap e782: 8 total, 8 up, 8 in 2024-03-30T14:29:49.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:49 smithi102 ceph-mon[29392]: osdmap e782: 8 total, 8 up, 8 in 2024-03-30T14:29:49.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:49 smithi102 ceph-mon[37717]: osdmap e782: 8 total, 8 up, 8 in 2024-03-30T14:29:50.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:50 smithi138 ceph-mon[33397]: pgmap v1685: 164 pgs: 164 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:50.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:50 smithi138 ceph-mon[33397]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:50.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:50 smithi138 ceph-mon[33397]: osdmap e783: 8 total, 8 up, 8 in 2024-03-30T14:29:50.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:50 smithi102 ceph-mon[29392]: pgmap v1685: 164 pgs: 164 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:50.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:50 smithi102 ceph-mon[29392]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:50.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:50 smithi102 ceph-mon[29392]: osdmap e783: 8 total, 8 up, 8 in 2024-03-30T14:29:50.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:50 smithi102 ceph-mon[37717]: pgmap v1685: 164 pgs: 164 active+clean; 582 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:50.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:50 smithi102 ceph-mon[37717]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:50.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:50 smithi102 ceph-mon[37717]: osdmap e783: 8 total, 8 up, 8 in 2024-03-30T14:29:51.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:51 smithi138 ceph-mon[33397]: osdmap e784: 8 total, 8 up, 8 in 2024-03-30T14:29:51.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:51 smithi102 ceph-mon[29392]: osdmap e784: 8 total, 8 up, 8 in 2024-03-30T14:29:51.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:51 smithi102 ceph-mon[37717]: osdmap e784: 8 total, 8 up, 8 in 2024-03-30T14:29:52.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:52 smithi138 ceph-mon[33397]: pgmap v1688: 196 pgs: 6 creating+activating, 26 creating+peering, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:52.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:52 smithi138 ceph-mon[33397]: osdmap e785: 8 total, 8 up, 8 in 2024-03-30T14:29:52.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:52 smithi102 ceph-mon[29392]: pgmap v1688: 196 pgs: 6 creating+activating, 26 creating+peering, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:52.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:52 smithi102 ceph-mon[29392]: osdmap e785: 8 total, 8 up, 8 in 2024-03-30T14:29:52.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:52 smithi102 ceph-mon[37717]: pgmap v1688: 196 pgs: 6 creating+activating, 26 creating+peering, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:52.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:52 smithi102 ceph-mon[37717]: osdmap e785: 8 total, 8 up, 8 in 2024-03-30T14:29:53.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:53 smithi138 ceph-mon[33397]: osdmap e786: 8 total, 8 up, 8 in 2024-03-30T14:29:53.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:53 smithi102 ceph-mon[29392]: osdmap e786: 8 total, 8 up, 8 in 2024-03-30T14:29:53.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:53 smithi102 ceph-mon[37717]: osdmap e786: 8 total, 8 up, 8 in 2024-03-30T14:29:54.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:54 smithi138 ceph-mon[33397]: pgmap v1691: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:54.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:54 smithi138 ceph-mon[33397]: osdmap e787: 8 total, 8 up, 8 in 2024-03-30T14:29:54.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:54 smithi102 ceph-mon[29392]: pgmap v1691: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:54.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:54 smithi102 ceph-mon[29392]: osdmap e787: 8 total, 8 up, 8 in 2024-03-30T14:29:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:54 smithi102 ceph-mon[37717]: pgmap v1691: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:54.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:54 smithi102 ceph-mon[37717]: osdmap e787: 8 total, 8 up, 8 in 2024-03-30T14:29:55.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:55 smithi138 ceph-mon[33397]: osdmap e788: 8 total, 8 up, 8 in 2024-03-30T14:29:55.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:55 smithi102 ceph-mon[29392]: osdmap e788: 8 total, 8 up, 8 in 2024-03-30T14:29:55.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:55 smithi102 ceph-mon[37717]: osdmap e788: 8 total, 8 up, 8 in 2024-03-30T14:29:56.589 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:29:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:29:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:29:56.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:56 smithi102 ceph-mon[29392]: pgmap v1694: 164 pgs: 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:29:56.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:56 smithi102 ceph-mon[29392]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:56.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:56 smithi102 ceph-mon[29392]: osdmap e789: 8 total, 8 up, 8 in 2024-03-30T14:29:56.589 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:56 smithi102 ceph-mon[37717]: pgmap v1694: 164 pgs: 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:29:56.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:56 smithi102 ceph-mon[37717]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:56.590 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:56 smithi102 ceph-mon[37717]: osdmap e789: 8 total, 8 up, 8 in 2024-03-30T14:29:56.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:56 smithi138 ceph-mon[33397]: pgmap v1694: 164 pgs: 164 active+clean; 582 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:29:56.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:56 smithi138 ceph-mon[33397]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:29:56.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:56 smithi138 ceph-mon[33397]: osdmap e789: 8 total, 8 up, 8 in 2024-03-30T14:29:57.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:57 smithi138 ceph-mon[33397]: osdmap e790: 8 total, 8 up, 8 in 2024-03-30T14:29:57.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:29:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:29:57.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:57 smithi102 ceph-mon[29392]: osdmap e790: 8 total, 8 up, 8 in 2024-03-30T14:29:57.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:57 smithi102 ceph-mon[37717]: osdmap e790: 8 total, 8 up, 8 in 2024-03-30T14:29:58.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:58 smithi138 ceph-mon[33397]: pgmap v1697: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:58.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:58 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:58.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:58 smithi138 ceph-mon[33397]: osdmap e791: 8 total, 8 up, 8 in 2024-03-30T14:29:58.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:58.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[29392]: pgmap v1697: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:58.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:58.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[29392]: osdmap e791: 8 total, 8 up, 8 in 2024-03-30T14:29:58.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:58.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[37717]: pgmap v1697: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:29:58.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:29:58.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[37717]: osdmap e791: 8 total, 8 up, 8 in 2024-03-30T14:29:58.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:29:59.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:29:59 smithi138 ceph-mon[33397]: osdmap e792: 8 total, 8 up, 8 in 2024-03-30T14:29:59.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:29:59 smithi102 ceph-mon[29392]: osdmap e792: 8 total, 8 up, 8 in 2024-03-30T14:29:59.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:29:59 smithi102 ceph-mon[37717]: osdmap e792: 8 total, 8 up, 8 in 2024-03-30T14:30:00.683 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:00 smithi138 ceph-mon[33397]: pgmap v1700: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:00.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:00 smithi138 ceph-mon[33397]: osdmap e793: 8 total, 8 up, 8 in 2024-03-30T14:30:00.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:00 smithi138 ceph-mon[33397]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-30T14:30:00.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:00 smithi102 ceph-mon[29392]: pgmap v1700: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:00.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:00 smithi102 ceph-mon[29392]: osdmap e793: 8 total, 8 up, 8 in 2024-03-30T14:30:00.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:00 smithi102 ceph-mon[29392]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-30T14:30:00.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:00 smithi102 ceph-mon[37717]: pgmap v1700: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:00.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:00 smithi102 ceph-mon[37717]: osdmap e793: 8 total, 8 up, 8 in 2024-03-30T14:30:00.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:00 smithi102 ceph-mon[37717]: overall HEALTH_WARN 2 pool(s) do not have an application enabled 2024-03-30T14:30:01.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:01 smithi138 ceph-mon[33397]: osdmap e794: 8 total, 8 up, 8 in 2024-03-30T14:30:01.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:01 smithi102 ceph-mon[29392]: osdmap e794: 8 total, 8 up, 8 in 2024-03-30T14:30:01.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:01 smithi102 ceph-mon[37717]: osdmap e794: 8 total, 8 up, 8 in 2024-03-30T14:30:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:02 smithi138 ceph-mon[33397]: pgmap v1703: 164 pgs: 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:02 smithi138 ceph-mon[33397]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:02.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:02 smithi138 ceph-mon[33397]: osdmap e795: 8 total, 8 up, 8 in 2024-03-30T14:30:02.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:02 smithi102 ceph-mon[29392]: pgmap v1703: 164 pgs: 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:02.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:02 smithi102 ceph-mon[29392]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:02.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:02 smithi102 ceph-mon[29392]: osdmap e795: 8 total, 8 up, 8 in 2024-03-30T14:30:02.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:02 smithi102 ceph-mon[37717]: pgmap v1703: 164 pgs: 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:02.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:02 smithi102 ceph-mon[37717]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:02.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:02 smithi102 ceph-mon[37717]: osdmap e795: 8 total, 8 up, 8 in 2024-03-30T14:30:03.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:03 smithi138 ceph-mon[33397]: osdmap e796: 8 total, 8 up, 8 in 2024-03-30T14:30:03.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:03 smithi138 ceph-mon[33397]: osdmap e797: 8 total, 8 up, 8 in 2024-03-30T14:30:03.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:03 smithi102 ceph-mon[29392]: osdmap e796: 8 total, 8 up, 8 in 2024-03-30T14:30:03.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:03 smithi102 ceph-mon[29392]: osdmap e797: 8 total, 8 up, 8 in 2024-03-30T14:30:03.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:03 smithi102 ceph-mon[37717]: osdmap e796: 8 total, 8 up, 8 in 2024-03-30T14:30:03.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:03 smithi102 ceph-mon[37717]: osdmap e797: 8 total, 8 up, 8 in 2024-03-30T14:30:04.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:04 smithi138 ceph-mon[33397]: pgmap v1706: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:04.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:04 smithi102 ceph-mon[29392]: pgmap v1706: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:04.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:04 smithi102 ceph-mon[37717]: pgmap v1706: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:05.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:05 smithi138 ceph-mon[33397]: osdmap e798: 8 total, 8 up, 8 in 2024-03-30T14:30:05.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:05 smithi102 ceph-mon[29392]: osdmap e798: 8 total, 8 up, 8 in 2024-03-30T14:30:05.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:05 smithi102 ceph-mon[37717]: osdmap e798: 8 total, 8 up, 8 in 2024-03-30T14:30:06.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:06 smithi138 ceph-mon[33397]: pgmap v1709: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:30:06.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:06 smithi138 ceph-mon[33397]: osdmap e799: 8 total, 8 up, 8 in 2024-03-30T14:30:06.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:06 smithi102 ceph-mon[29392]: pgmap v1709: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:30:06.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:06 smithi102 ceph-mon[29392]: osdmap e799: 8 total, 8 up, 8 in 2024-03-30T14:30:06.688 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:30:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:30:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:30:06.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:06 smithi102 ceph-mon[37717]: pgmap v1709: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:30:06.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:06 smithi102 ceph-mon[37717]: osdmap e799: 8 total, 8 up, 8 in 2024-03-30T14:30:07.357 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:30:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:30:07.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:07 smithi138 ceph-mon[33397]: osdmap e800: 8 total, 8 up, 8 in 2024-03-30T14:30:07.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:07 smithi138 ceph-mon[33397]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:07.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:07 smithi102 ceph-mon[29392]: osdmap e800: 8 total, 8 up, 8 in 2024-03-30T14:30:07.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:07 smithi102 ceph-mon[29392]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:07.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:07 smithi102 ceph-mon[37717]: osdmap e800: 8 total, 8 up, 8 in 2024-03-30T14:30:07.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:07 smithi102 ceph-mon[37717]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:08.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:08 smithi138 ceph-mon[33397]: pgmap v1712: 164 pgs: 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:08.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:08 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:08.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:08 smithi138 ceph-mon[33397]: osdmap e801: 8 total, 8 up, 8 in 2024-03-30T14:30:08.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:08 smithi102 ceph-mon[29392]: pgmap v1712: 164 pgs: 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:08.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:08 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:08.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:08 smithi102 ceph-mon[29392]: osdmap e801: 8 total, 8 up, 8 in 2024-03-30T14:30:08.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:08 smithi102 ceph-mon[37717]: pgmap v1712: 164 pgs: 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:08.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:08 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:08.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:08 smithi102 ceph-mon[37717]: osdmap e801: 8 total, 8 up, 8 in 2024-03-30T14:30:09.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:09 smithi138 ceph-mon[33397]: osdmap e802: 8 total, 8 up, 8 in 2024-03-30T14:30:09.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:09 smithi102 ceph-mon[29392]: osdmap e802: 8 total, 8 up, 8 in 2024-03-30T14:30:09.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:09 smithi102 ceph-mon[37717]: osdmap e802: 8 total, 8 up, 8 in 2024-03-30T14:30:10.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:10 smithi138 ceph-mon[33397]: pgmap v1715: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:10.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:10 smithi138 ceph-mon[33397]: osdmap e803: 8 total, 8 up, 8 in 2024-03-30T14:30:10.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:10 smithi102 ceph-mon[29392]: pgmap v1715: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:10.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:10 smithi102 ceph-mon[29392]: osdmap e803: 8 total, 8 up, 8 in 2024-03-30T14:30:10.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:10 smithi102 ceph-mon[37717]: pgmap v1715: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:10.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:10 smithi102 ceph-mon[37717]: osdmap e803: 8 total, 8 up, 8 in 2024-03-30T14:30:11.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:11 smithi138 ceph-mon[33397]: osdmap e804: 8 total, 8 up, 8 in 2024-03-30T14:30:11.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:11 smithi102 ceph-mon[29392]: osdmap e804: 8 total, 8 up, 8 in 2024-03-30T14:30:11.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:11 smithi102 ceph-mon[37717]: osdmap e804: 8 total, 8 up, 8 in 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: Running main() from gmock_main.cc 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [----------] Global test environment set-up. 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (1809426 ms) 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3024 ms) 2024-03-30T14:30:12.404 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3025 ms) 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3015 ms) 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3030 ms) 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3030 ms) 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3018 ms) 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3036 ms) 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-30T14:30:12.405 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3017 ms) 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3016 ms) 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3011 ms) 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3041 ms) 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3023 ms) 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-30T14:30:12.406 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3020 ms) 2024-03-30T14:30:12.407 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (1848732 ms total) 2024-03-30T14:30:12.407 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: 2024-03-30T14:30:12.407 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [----------] Global test environment tear-down 2024-03-30T14:30:12.407 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (1848732 ms total) 2024-03-30T14:30:12.407 INFO:tasks.workunit.client.0.smithi102.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75090 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75090 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75636 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75636 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75901 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75901 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74987 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74987 2024-03-30T14:30:12.411 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75527 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75527 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75763 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75763 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75575 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75575 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75831 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75831 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75384 2024-03-30T14:30:12.412 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75384 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74908 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74908 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75877 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75877 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75296 2024-03-30T14:30:12.413 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75296 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74736 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74736 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74795 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74795 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.414 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75219 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75219 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=74833 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 74833 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75323 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75323 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75361 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75361 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.415 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75745 2024-03-30T14:30:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75745 2024-03-30T14:30:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:30:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=75946 2024-03-30T14:30:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 75946 2024-03-30T14:30:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:+ exit 0 2024-03-30T14:30:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:+ cleanup 2024-03-30T14:30:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:+ pkill -P 74730 2024-03-30T14:30:12.431 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-03-30T14:30:12.431 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T14:30:12.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T14:30:12.465 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-30T14:30:12.466 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-30T14:30:12.466 DEBUG:teuthology.orchestra.run.smithi102:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5af0400050a123dc31f105425174f106e057276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-30T14:30:12.525 INFO:tasks.workunit.client.0.smithi102.stderr:++ uuidgen 2024-03-30T14:30:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:+ p=b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:30:12.528 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool create b802dd6e-da88-4515-b4f3-9f10759279d2 12 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1869056667 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 msgr2=0x7fdd4810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1869056667 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd4810b7c0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009900 tx=0x7fdd3c02bcd0 comp rx=0 tx=0).stop 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1869056667 shutdown_connections 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1869056667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd4810bd90 0x7fdd4810e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1869056667 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd4810b7c0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1869056667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd481029c0 0x7fdd48102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1869056667 >> 172.21.15.102:0/1869056667 conn(0x7fdd48077740 msgr2=0x7fdd48077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1869056667 shutdown_connections 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1869056667 wait complete. 2024-03-30T14:30:12.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.622+0000 7fdd4cc64640 1 Processor -- start 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 -- start start 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd481029c0 0x7fdd48100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd48100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd4810bd90 0x7fdd48101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48111230 con 0x7fdd4810bd90 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd481110b0 con 0x7fdd481029c0 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdd481113b0 con 0x7fdd481032e0 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd45d9b640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd48100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd46d9d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd4810bd90 0x7fdd48101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd45d9b640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd48100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:58624/0 (socket says 172.21.15.102:58624) 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd45d9b640 1 -- 172.21.15.102:0/1627043971 learned_addr learned my addr 172.21.15.102:0/1627043971 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4659c640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd481029c0 0x7fdd48100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:12.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd46d9d640 1 -- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 msgr2=0x7fdd48100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd46d9d640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd48100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd46d9d640 1 -- 172.21.15.102:0/1627043971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd481029c0 msgr2=0x7fdd48100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd46d9d640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd481029c0 0x7fdd48100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd46d9d640 1 -- 172.21.15.102:0/1627043971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd481a5c40 con 0x7fdd4810bd90 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd45d9b640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd48100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd46d9d640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd4810bd90 0x7fdd48101090 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000ca00 tx=0x7fdd4000cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd40007d30 con 0x7fdd4810bd90 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd481a5f30 con 0x7fdd4810bd90 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fdd40005020 con 0x7fdd4810bd90 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000f690 con 0x7fdd4810bd90 2024-03-30T14:30:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.623+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd481a6400 con 0x7fdd4810bd90 2024-03-30T14:30:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.624+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd4806a6b0 con 0x7fdd4810bd90 2024-03-30T14:30:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.625+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fdd40004a40 con 0x7fdd4810bd90 2024-03-30T14:30:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.625+0000 7fdd377fe640 1 --2- 172.21.15.102:0/1627043971 >> v2:172.21.15.102:6800/622906077 conn(0x7fdd3006db70 0x7fdd30070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.626+0000 7fdd4659c640 1 --2- 172.21.15.102:0/1627043971 >> v2:172.21.15.102:6800/622906077 conn(0x7fdd3006db70 0x7fdd30070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.626+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(806..806 src has 266..806) ==== 7418+0+0 (secure 0 0 0) 0x7fdd4008e1d0 con 0x7fdd4810bd90 2024-03-30T14:30:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.626+0000 7fdd4659c640 1 --2- 172.21.15.102:0/1627043971 >> v2:172.21.15.102:6800/622906077 conn(0x7fdd3006db70 0x7fdd30070030 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fdd380095f0 tx=0x7fdd38005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:12.626 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.630+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdd40014070 con 0x7fdd4810bd90 2024-03-30T14:30:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:12 smithi138 ceph-mon[33397]: pgmap v1718: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:12.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:12 smithi138 ceph-mon[33397]: osdmap e805: 8 total, 8 up, 8 in 2024-03-30T14:30:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:12 smithi102 ceph-mon[29392]: pgmap v1718: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:12.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:12 smithi102 ceph-mon[29392]: osdmap e805: 8 total, 8 up, 8 in 2024-03-30T14:30:12.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:12 smithi102 ceph-mon[37717]: pgmap v1718: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:12.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:12 smithi102 ceph-mon[37717]: osdmap e805: 8 total, 8 up, 8 in 2024-03-30T14:30:12.802 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:12.806+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12} v 0) -- 0x7fdd48068a80 con 0x7fdd4810bd90 2024-03-30T14:30:13.398 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.402+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]=0 pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' created v807) ==== 176+0+0 (secure 0 0 0) 0x7fdd4005f280 con 0x7fdd4810bd90 2024-03-30T14:30:13.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.513+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12} v 0) -- 0x7fdd48101b40 con 0x7fdd4810bd90 2024-03-30T14:30:13.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.514+0000 7fdd377fe640 1 -- 172.21.15.102:0/1627043971 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]=0 pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' already exists v807) ==== 183+0+0 (secure 0 0 0) 0x7fdd40063ab0 con 0x7fdd4810bd90 2024-03-30T14:30:13.510 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' already exists 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 >> v2:172.21.15.102:6800/622906077 conn(0x7fdd3006db70 msgr2=0x7fdd30070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1627043971 >> v2:172.21.15.102:6800/622906077 conn(0x7fdd3006db70 0x7fdd30070030 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fdd380095f0 tx=0x7fdd38005f90 comp rx=0 tx=0).stop 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd4810bd90 msgr2=0x7fdd48101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd4810bd90 0x7fdd48101090 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000ca00 tx=0x7fdd4000cec0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 shutdown_connections 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1627043971 >> v2:172.21.15.102:6800/622906077 conn(0x7fdd3006db70 0x7fdd30070030 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd4810bd90 0x7fdd48101090 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdd481032e0 0x7fdd48100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 --2- 172.21.15.102:0/1627043971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd481029c0 0x7fdd48100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 >> 172.21.15.102:0/1627043971 conn(0x7fdd48077740 msgr2=0x7fdd480fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:13.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 shutdown_connections 2024-03-30T14:30:13.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.516+0000 7fdd4cc64640 1 -- 172.21.15.102:0/1627043971 wait complete. 2024-03-30T14:30:13.521 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota b802dd6e-da88-4515-b4f3-9f10759279d2 max_objects 10 2024-03-30T14:30:13.608 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 -- 172.21.15.102:0/1481499517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100fd0 msgr2=0x7fa25010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:13.608 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 --2- 172.21.15.102:0/1481499517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100fd0 0x7fa25010d4e0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7fa248009c10 tx=0x7fa24802bef0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.608 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 -- 172.21.15.102:0/1481499517 shutdown_connections 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 --2- 172.21.15.102:0/1481499517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100fd0 0x7fa25010d4e0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 --2- 172.21.15.102:0/1481499517 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa250100610 0x7fa250100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 --2- 172.21.15.102:0/1481499517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2501066b0 0x7fa250106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 -- 172.21.15.102:0/1481499517 >> 172.21.15.102:0/1481499517 conn(0x7fa250077770 msgr2=0x7fa250077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 -- 172.21.15.102:0/1481499517 shutdown_connections 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 -- 172.21.15.102:0/1481499517 wait complete. 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 Processor -- start 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.610+0000 7fa256756640 1 -- start start 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa256756640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100610 0x7fa2501964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa256756640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa250100fd0 0x7fa2501969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa256756640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2501066b0 0x7fa25019ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa256756640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa250110100 con 0x7fa2501066b0 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa256756640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010ff80 con 0x7fa250100fd0 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa256756640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa250110280 con 0x7fa250100610 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100610 0x7fa2501964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa254cf3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2501066b0 0x7fa25019ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100610 0x7fa2501964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:58638/0 (socket says 172.21.15.102:58638) 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 -- 172.21.15.102:0/3938524864 learned_addr learned my addr 172.21.15.102:0/3938524864 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24f7fe640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa250100fd0 0x7fa2501969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 -- 172.21.15.102:0/3938524864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa250100fd0 msgr2=0x7fa2501969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa250100fd0 0x7fa2501969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 -- 172.21.15.102:0/3938524864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2501066b0 msgr2=0x7fa25019ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2501066b0 0x7fa25019ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 -- 172.21.15.102:0/3938524864 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25019b490 con 0x7fa250100610 2024-03-30T14:30:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24ffff640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100610 0x7fa2501964a0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fa24000e9a0 tx=0x7fa24000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.611+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24000cd80 con 0x7fa250100610 2024-03-30T14:30:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.612+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25019b780 con 0x7fa250100610 2024-03-30T14:30:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.612+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fa2400108e0 con 0x7fa250100610 2024-03-30T14:30:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.612+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2400188f0 con 0x7fa250100610 2024-03-30T14:30:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.612+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2501a3010 con 0x7fa250100610 2024-03-30T14:30:13.611 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.612+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa250068630 con 0x7fa250100610 2024-03-30T14:30:13.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.613+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fa240010430 con 0x7fa250100610 2024-03-30T14:30:13.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.613+0000 7fa24d7fa640 1 --2- 172.21.15.102:0/3938524864 >> v2:172.21.15.102:6800/622906077 conn(0x7fa23806db70 0x7fa238070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:13.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.614+0000 7fa24f7fe640 1 --2- 172.21.15.102:0/3938524864 >> v2:172.21.15.102:6800/622906077 conn(0x7fa23806db70 0x7fa238070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:13.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.614+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(807..807 src has 266..807) ==== 7793+0+0 (secure 0 0 0) 0x7fa24008e540 con 0x7fa250100610 2024-03-30T14:30:13.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.614+0000 7fa24f7fe640 1 --2- 172.21.15.102:0/3938524864 >> v2:172.21.15.102:6800/622906077 conn(0x7fa23806db70 0x7fa238070030 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fa244005e10 tx=0x7fa244005da0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:13.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.618+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa240053280 con 0x7fa250100610 2024-03-30T14:30:13.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:13 smithi138 ceph-mon[33397]: osdmap e806: 8 total, 8 up, 8 in 2024-03-30T14:30:13.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:13 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]: dispatch 2024-03-30T14:30:13.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:13.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:13 smithi102 ceph-mon[29392]: osdmap e806: 8 total, 8 up, 8 in 2024-03-30T14:30:13.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:13 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]: dispatch 2024-03-30T14:30:13.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:13.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:13 smithi102 ceph-mon[37717]: osdmap e806: 8 total, 8 up, 8 in 2024-03-30T14:30:13.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:13 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]: dispatch 2024-03-30T14:30:13.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:13.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:13.794+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"} v 0) -- 0x7fa2501a3240 con 0x7fa250100610 2024-03-30T14:30:14.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:14.407+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v808) ==== 223+0+0 (secure 0 0 0) 0x7fa24005ba20 con 0x7fa250100610 2024-03-30T14:30:14.520 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:14.524+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"} v 0) -- 0x7fa2501977f0 con 0x7fa250100610 2024-03-30T14:30:14.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:14 smithi138 ceph-mon[33397]: pgmap v1721: 164 pgs: 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:14.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:14 smithi138 ceph-mon[33397]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:14.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]': finished 2024-03-30T14:30:14.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:14 smithi138 ceph-mon[33397]: osdmap e807: 8 total, 8 up, 8 in 2024-03-30T14:30:14.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]: dispatch 2024-03-30T14:30:14.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:14 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3938524864' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:14.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:14 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[29392]: pgmap v1721: 164 pgs: 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[29392]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]': finished 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[29392]: osdmap e807: 8 total, 8 up, 8 in 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]: dispatch 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3938524864' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[37717]: pgmap v1721: 164 pgs: 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[37717]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]': finished 2024-03-30T14:30:14.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[37717]: osdmap e807: 8 total, 8 up, 8 in 2024-03-30T14:30:14.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1627043971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pg_num": 12}]: dispatch 2024-03-30T14:30:14.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3938524864' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:14.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:14 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:15.424 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.428+0000 7fa24d7fa640 1 -- 172.21.15.102:0/3938524864 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v809) ==== 223+0+0 (secure 0 0 0) 0x7fa24005f470 con 0x7fa250100610 2024-03-30T14:30:15.424 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_objects = 10 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.430+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 >> v2:172.21.15.102:6800/622906077 conn(0x7fa23806db70 msgr2=0x7fa238070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.430+0000 7fa256756640 1 --2- 172.21.15.102:0/3938524864 >> v2:172.21.15.102:6800/622906077 conn(0x7fa23806db70 0x7fa238070030 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fa244005e10 tx=0x7fa244005da0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.430+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100610 msgr2=0x7fa2501964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.430+0000 7fa256756640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100610 0x7fa2501964a0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fa24000e9a0 tx=0x7fa24000ee60 comp rx=0 tx=0).stop 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 shutdown_connections 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 --2- 172.21.15.102:0/3938524864 >> v2:172.21.15.102:6800/622906077 conn(0x7fa23806db70 0x7fa238070030 unknown :-1 s=CLOSED pgs=3466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2501066b0 0x7fa25019ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa250100fd0 0x7fa2501969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 --2- 172.21.15.102:0/3938524864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa250100610 0x7fa2501964a0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 >> 172.21.15.102:0/3938524864 conn(0x7fa250077770 msgr2=0x7fa2500fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 shutdown_connections 2024-03-30T14:30:15.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.431+0000 7fa256756640 1 -- 172.21.15.102:0/3938524864 wait complete. 2024-03-30T14:30:15.438 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool application enable b802dd6e-da88-4515-b4f3-9f10759279d2 rados 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.529+0000 7f9931ba0640 1 -- 172.21.15.102:0/1673442805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 msgr2=0x7f992c107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.529+0000 7f9931ba0640 1 --2- 172.21.15.102:0/1673442805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 0x7f992c107af0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f991c009930 tx=0x7f991c02bc30 comp rx=0 tx=0).stop 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.529+0000 7f9931ba0640 1 -- 172.21.15.102:0/1673442805 shutdown_connections 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.529+0000 7f9931ba0640 1 --2- 172.21.15.102:0/1673442805 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c069420 0x7f992c10e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.529+0000 7f9931ba0640 1 --2- 172.21.15.102:0/1673442805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f992c068a80 0x7f992c068ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 --2- 172.21.15.102:0/1673442805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 0x7f992c107af0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 -- 172.21.15.102:0/1673442805 >> 172.21.15.102:0/1673442805 conn(0x7f992c075fb0 msgr2=0x7f992c0763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:15.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 -- 172.21.15.102:0/1673442805 shutdown_connections 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 -- 172.21.15.102:0/1673442805 wait complete. 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 Processor -- start 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 -- start start 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c068a80 0x7f992c197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f992c069420 0x7f992c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 0x7f992c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f992c111030 con 0x7f992c107710 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f992c110eb0 con 0x7f992c069420 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9931ba0640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f992c1111b0 con 0x7f992c068a80 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9930b9e640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c068a80 0x7f992c197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f993139f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 0x7f992c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9930b9e640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c068a80 0x7f992c197470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:58652/0 (socket says 172.21.15.102:58652) 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.530+0000 7f9930b9e640 1 -- 172.21.15.102:0/3345238164 learned_addr learned my addr 172.21.15.102:0/3345238164 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:30:15.529 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f992bfff640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f992c069420 0x7f992c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9930b9e640 1 -- 172.21.15.102:0/3345238164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f992c069420 msgr2=0x7f992c1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9930b9e640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f992c069420 0x7f992c1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9930b9e640 1 -- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 msgr2=0x7f992c19bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9930b9e640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 0x7f992c19bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9930b9e640 1 -- 172.21.15.102:0/3345238164 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992c19c460 con 0x7f992c068a80 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f993139f640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 0x7f992c19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9930b9e640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c068a80 0x7f992c197470 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f991c009230 tx=0x7f991c02c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c043020 con 0x7f992c068a80 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f991c0043d0 con 0x7f992c068a80 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f992c19c6f0 con 0x7f992c068a80 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c003b70 con 0x7f992c068a80 2024-03-30T14:30:15.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.531+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f992c1a3fe0 con 0x7f992c068a80 2024-03-30T14:30:15.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.532+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f992c04f4b0 con 0x7f992c068a80 2024-03-30T14:30:15.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.533+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f991c046020 con 0x7f992c068a80 2024-03-30T14:30:15.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.533+0000 7f9929ffb640 1 --2- 172.21.15.102:0/3345238164 >> v2:172.21.15.102:6800/622906077 conn(0x7f991406dc10 0x7f99140700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:15.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.533+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(809..809 src has 266..809) ==== 7793+0+0 (secure 0 0 0) 0x7f991c0afcc0 con 0x7f992c068a80 2024-03-30T14:30:15.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.533+0000 7f992bfff640 1 --2- 172.21.15.102:0/3345238164 >> v2:172.21.15.102:6800/622906077 conn(0x7f991406dc10 0x7f99140700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:15.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.534+0000 7f992bfff640 1 --2- 172.21.15.102:0/3345238164 >> v2:172.21.15.102:6800/622906077 conn(0x7f991406dc10 0x7f99140700d0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f992c111af0 tx=0x7f9920008040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:15.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.538+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f991c04d070 con 0x7f992c068a80 2024-03-30T14:30:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:30:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:15 smithi138 ceph-mon[33397]: osdmap e808: 8 total, 8 up, 8 in 2024-03-30T14:30:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:15 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3938524864' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:15.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:15 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:15.687 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:30:15.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[29392]: osdmap e808: 8 total, 8 up, 8 in 2024-03-30T14:30:15.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3938524864' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:15.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:15.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:30:15.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[37717]: osdmap e808: 8 total, 8 up, 8 in 2024-03-30T14:30:15.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3938524864' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:15.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:15 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:30:15.711 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:15.715+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"} v 0) -- 0x7f992c1a41b0 con 0x7f992c068a80 2024-03-30T14:30:16.423 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:16.427+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]=0 enabled application 'rados' on pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' v810) ==== 213+0+0 (secure 0 0 0) 0x7f991c07d1a0 con 0x7f992c068a80 2024-03-30T14:30:16.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:16.537+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"} v 0) -- 0x7f992c061d30 con 0x7f992c068a80 2024-03-30T14:30:16.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:16 smithi138 ceph-mon[33397]: pgmap v1724: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:30:16.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:30:16.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:16 smithi138 ceph-mon[33397]: osdmap e809: 8 total, 8 up, 8 in 2024-03-30T14:30:16.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:16 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3345238164' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:16.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:16 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:16.687 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:30:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[37717]: pgmap v1724: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[37717]: osdmap e809: 8 total, 8 up, 8 in 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3345238164' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[29392]: pgmap v1724: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 816 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[29392]: osdmap e809: 8 total, 8 up, 8 in 2024-03-30T14:30:16.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3345238164' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:16.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:16 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:17.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:30:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:30:17.435 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.439+0000 7f9929ffb640 1 -- 172.21.15.102:0/3345238164 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]=0 enabled application 'rados' on pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' v811) ==== 213+0+0 (secure 0 0 0) 0x7f991c080bf0 con 0x7f992c068a80 2024-03-30T14:30:17.435 INFO:tasks.workunit.client.0.smithi102.stderr:enabled application 'rados' on pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' 2024-03-30T14:30:17.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 >> v2:172.21.15.102:6800/622906077 conn(0x7f991406dc10 msgr2=0x7f99140700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:17.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 --2- 172.21.15.102:0/3345238164 >> v2:172.21.15.102:6800/622906077 conn(0x7f991406dc10 0x7f99140700d0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f992c111af0 tx=0x7f9920008040 comp rx=0 tx=0).stop 2024-03-30T14:30:17.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c068a80 msgr2=0x7f992c197470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:17.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c068a80 0x7f992c197470 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f991c009230 tx=0x7f991c02c790 comp rx=0 tx=0).stop 2024-03-30T14:30:17.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 shutdown_connections 2024-03-30T14:30:17.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 --2- 172.21.15.102:0/3345238164 >> v2:172.21.15.102:6800/622906077 conn(0x7f991406dc10 0x7f99140700d0 unknown :-1 s=CLOSED pgs=3467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:17.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f992c107710 0x7f992c19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:17.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f992c069420 0x7f992c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:17.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 --2- 172.21.15.102:0/3345238164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f992c068a80 0x7f992c197470 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:17.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 >> 172.21.15.102:0/3345238164 conn(0x7f992c075fb0 msgr2=0x7f992c105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:17.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 shutdown_connections 2024-03-30T14:30:17.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:17.442+0000 7f9931ba0640 1 -- 172.21.15.102:0/3345238164 wait complete. 2024-03-30T14:30:17.449 INFO:tasks.workunit.client.0.smithi102.stderr:++ seq 1 10 2024-03-30T14:30:17.468 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.468 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj1 /etc/passwd 2024-03-30T14:30:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj2 /etc/passwd 2024-03-30T14:30:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.544 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj3 /etc/passwd 2024-03-30T14:30:17.579 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.579 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj4 /etc/passwd 2024-03-30T14:30:17.615 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.615 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj5 /etc/passwd 2024-03-30T14:30:17.652 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.652 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj6 /etc/passwd 2024-03-30T14:30:17.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]': finished 2024-03-30T14:30:17.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:17 smithi138 ceph-mon[33397]: osdmap e810: 8 total, 8 up, 8 in 2024-03-30T14:30:17.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:17 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3345238164' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:17.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:17 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:17.686 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.686 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj7 /etc/passwd 2024-03-30T14:30:17.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]': finished 2024-03-30T14:30:17.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[37717]: osdmap e810: 8 total, 8 up, 8 in 2024-03-30T14:30:17.687 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3345238164' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:17.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:17.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]': finished 2024-03-30T14:30:17.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[29392]: osdmap e810: 8 total, 8 up, 8 in 2024-03-30T14:30:17.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3345238164' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:17.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:17 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]: dispatch 2024-03-30T14:30:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.722 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj8 /etc/passwd 2024-03-30T14:30:17.756 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.756 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj9 /etc/passwd 2024-03-30T14:30:17.790 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:30:17.790 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put obj10 /etc/passwd 2024-03-30T14:30:17.824 INFO:tasks.workunit.client.0.smithi102.stderr:+ sleep 30 2024-03-30T14:30:18.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:18 smithi138 ceph-mon[33397]: pgmap v1727: 176 pgs: 176 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:18.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:18 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:18.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:18 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]': finished 2024-03-30T14:30:18.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:18 smithi138 ceph-mon[33397]: osdmap e811: 8 total, 8 up, 8 in 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[29392]: pgmap v1727: 176 pgs: 176 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]': finished 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[29392]: osdmap e811: 8 total, 8 up, 8 in 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[37717]: pgmap v1727: 176 pgs: 176 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "app": "rados"}]': finished 2024-03-30T14:30:18.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:18 smithi102 ceph-mon[37717]: osdmap e811: 8 total, 8 up, 8 in 2024-03-30T14:30:20.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:20 smithi102 ceph-mon[29392]: pgmap v1729: 176 pgs: 176 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:30:20.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:20 smithi102 ceph-mon[37717]: pgmap v1729: 176 pgs: 176 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:30:20.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:20 smithi138 ceph-mon[33397]: pgmap v1729: 176 pgs: 176 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:30:22.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[29392]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 3 op/s 2024-03-30T14:30:22.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[29392]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' is full (reached quota's max_objects: 10) 2024-03-30T14:30:22.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[29392]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:30:22.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[29392]: osdmap e812: 8 total, 8 up, 8 in 2024-03-30T14:30:22.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[37717]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 3 op/s 2024-03-30T14:30:22.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[37717]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' is full (reached quota's max_objects: 10) 2024-03-30T14:30:22.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[37717]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:30:22.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:22 smithi102 ceph-mon[37717]: osdmap e812: 8 total, 8 up, 8 in 2024-03-30T14:30:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:22 smithi138 ceph-mon[33397]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 20 KiB/s wr, 3 op/s 2024-03-30T14:30:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:22 smithi138 ceph-mon[33397]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' is full (reached quota's max_objects: 10) 2024-03-30T14:30:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:22 smithi138 ceph-mon[33397]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:30:22.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:22 smithi138 ceph-mon[33397]: osdmap e812: 8 total, 8 up, 8 in 2024-03-30T14:30:24.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:24 smithi138 ceph-mon[33397]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-30T14:30:24.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:24 smithi102 ceph-mon[29392]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-30T14:30:24.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:24 smithi102 ceph-mon[37717]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 19 KiB/s wr, 2 op/s 2024-03-30T14:30:26.740 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:26 smithi102 ceph-mon[29392]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-30T14:30:26.741 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:26 smithi102 ceph-mon[37717]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-30T14:30:26.742 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:30:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:30:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:30:26.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:26 smithi138 ceph-mon[33397]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-30T14:30:27.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:30:27 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:30:28.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:28 smithi138 ceph-mon[33397]: pgmap v1734: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-30T14:30:28.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:28 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:28.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:28 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:28.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:28.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[29392]: pgmap v1734: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-30T14:30:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:28.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:28.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[37717]: pgmap v1734: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-03-30T14:30:28.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:28.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:28.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:30.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:30 smithi138 ceph-mon[33397]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:30:30.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:30 smithi102 ceph-mon[37717]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:30:30.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:30 smithi102 ceph-mon[29392]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:30:32.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:32 smithi138 ceph-mon[33397]: pgmap v1736: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:30:32.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:32 smithi102 ceph-mon[37717]: pgmap v1736: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:30:32.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:32 smithi102 ceph-mon[29392]: pgmap v1736: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:30:34.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:34 smithi138 ceph-mon[33397]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-03-30T14:30:34.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:34 smithi102 ceph-mon[29392]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-03-30T14:30:34.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:34 smithi102 ceph-mon[37717]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-03-30T14:30:36.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:36 smithi138 ceph-mon[33397]: pgmap v1738: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:36.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:36 smithi102 ceph-mon[29392]: pgmap v1738: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:36.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:36 smithi102 ceph-mon[37717]: pgmap v1738: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:36.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:30:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:30:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:30:37.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:30:37 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:30:38.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:38 smithi138 ceph-mon[33397]: pgmap v1739: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:38.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:38 smithi102 ceph-mon[29392]: pgmap v1739: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:38.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:38.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:38 smithi102 ceph-mon[37717]: pgmap v1739: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:38.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:40.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:40 smithi138 ceph-mon[33397]: pgmap v1740: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:40.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:40 smithi102 ceph-mon[29392]: pgmap v1740: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:40.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:40 smithi102 ceph-mon[37717]: pgmap v1740: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:42.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:42 smithi138 ceph-mon[33397]: pgmap v1741: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:42.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:42 smithi102 ceph-mon[29392]: pgmap v1741: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:42.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:42 smithi102 ceph-mon[37717]: pgmap v1741: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:43.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:43.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:43.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:44.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:44 smithi138 ceph-mon[33397]: pgmap v1742: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:44.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:44 smithi102 ceph-mon[29392]: pgmap v1742: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:44.938 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:44 smithi102 ceph-mon[37717]: pgmap v1742: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:45.754 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:45 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:45.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:45 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:45.774 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:45 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:30:46.753 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:46 smithi138 ceph-mon[33397]: pgmap v1743: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:46.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:46 smithi102 ceph-mon[29392]: pgmap v1743: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:46.772 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:46 smithi102 ceph-mon[37717]: pgmap v1743: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:46.772 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:30:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:30:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:30:47.386 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:30:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:30:47.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:47 smithi138 ceph-mon[33397]: pgmap v1744: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:47.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:47 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:47.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:47 smithi102 ceph-mon[37717]: pgmap v1744: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:47.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:47 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:47.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:47 smithi102 ceph-mon[29392]: pgmap v1744: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:47.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:47 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:47.826 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=151644 2024-03-30T14:30:47.826 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota b802dd6e-da88-4515-b4f3-9f10759279d2 max_objects 100 2024-03-30T14:30:47.826 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put onemore /etc/passwd 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 -- 172.21.15.102:0/2399331298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 msgr2=0x7fa4b410d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 --2- 172.21.15.102:0/2399331298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b410d500 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009c10 tx=0x7fa4ac02bef0 comp rx=0 tx=0).stop 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 -- 172.21.15.102:0/2399331298 shutdown_connections 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 --2- 172.21.15.102:0/2399331298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b410d500 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 --2- 172.21.15.102:0/2399331298 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4b4100630 0x7fa4b4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 --2- 172.21.15.102:0/2399331298 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4b4106640 0x7fa4b4106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 -- 172.21.15.102:0/2399331298 >> 172.21.15.102:0/2399331298 conn(0x7fa4b4077740 msgr2=0x7fa4b4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 -- 172.21.15.102:0/2399331298 shutdown_connections 2024-03-30T14:30:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 -- 172.21.15.102:0/2399331298 wait complete. 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.929+0000 7fa4b971e640 1 Processor -- start 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b971e640 1 -- start start 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b971e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4b4100630 0x7fa4b4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b971e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b4198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b971e640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4b4106640 0x7fa4b41a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b971e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4110120 con 0x7fa4b4100ff0 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b971e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b410ffa0 con 0x7fa4b4100630 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b971e640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b41102a0 con 0x7fa4b4106640 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b4198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b4198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43424/0 (socket says 172.21.15.102:43424) 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b37fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4b4106640 0x7fa4b41a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 -- 172.21.15.102:0/453336625 learned_addr learned my addr 172.21.15.102:0/453336625 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:30:47.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b2ffd640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4b4100630 0x7fa4b4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 -- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4b4106640 msgr2=0x7fa4b41a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4b4106640 0x7fa4b41a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 -- 172.21.15.102:0/453336625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4b4100630 msgr2=0x7fa4b4198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4b4100630 0x7fa4b4198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b41a3b20 con 0x7fa4b4100ff0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b2ffd640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4b4100630 0x7fa4b4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b37fe640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4b4106640 0x7fa4b41a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.930+0000 7fa4b27fc640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b4198aa0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400e9c0 tx=0x7fa4a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.931+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400ccc0 con 0x7fa4b4100ff0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.931+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fa4a40054d0 con 0x7fa4b4100ff0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.931+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b41a3cf0 con 0x7fa4b4100ff0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.931+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4010430 con 0x7fa4b4100ff0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.931+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b41a41e0 con 0x7fa4b4100ff0 2024-03-30T14:30:47.929 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.932+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b4067da0 con 0x7fa4b4100ff0 2024-03-30T14:30:47.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.932+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fa4a4004f40 con 0x7fa4b4100ff0 2024-03-30T14:30:47.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.932+0000 7fa49bfff640 1 --2- 172.21.15.102:0/453336625 >> v2:172.21.15.102:6800/622906077 conn(0x7fa49c06db70 0x7fa49c070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:47.932 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.932+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(812..812 src has 266..812) ==== 7806+0+0 (secure 0 0 0) 0x7fa4a408e540 con 0x7fa4b4100ff0 2024-03-30T14:30:47.932 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.932+0000 7fa4b2ffd640 1 --2- 172.21.15.102:0/453336625 >> v2:172.21.15.102:6800/622906077 conn(0x7fa49c06db70 0x7fa49c070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:47.932 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.933+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=813}) -- 0x7fa49c079060 con 0x7fa4b4100ff0 2024-03-30T14:30:47.932 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.933+0000 7fa4b2ffd640 1 --2- 172.21.15.102:0/453336625 >> v2:172.21.15.102:6800/622906077 conn(0x7fa49c06db70 0x7fa49c070030 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80095f0 tx=0x7fa4a8005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:47.933 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:47.937+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa4a405f470 con 0x7fa4b4100ff0 2024-03-30T14:30:48.114 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:48.119+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"} v 0) -- 0x7fa4b4108ba0 con 0x7fa4b4100ff0 2024-03-30T14:30:48.851 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:48.855+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v813) ==== 225+0+0 (secure 0 0 0) 0x7fa4a4063ca0 con 0x7fa4b4100ff0 2024-03-30T14:30:48.853 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:48.858+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 8 ==== osd_map(813..813 src has 266..813) ==== 628+0+0 (secure 0 0 0) 0x7fa4a408c350 con 0x7fa4b4100ff0 2024-03-30T14:30:48.853 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:48.858+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=814}) -- 0x7fa49c079fc0 con 0x7fa4b4100ff0 2024-03-30T14:30:48.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:48 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:48.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:48 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:48.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:48 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]: dispatch 2024-03-30T14:30:48.958 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:48.963+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"} v 0) -- 0x7fa4b4108610 con 0x7fa4b4100ff0 2024-03-30T14:30:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:48 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:48 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:49.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:48 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]: dispatch 2024-03-30T14:30:49.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:48 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:49.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:48 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:49.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:48 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]: dispatch 2024-03-30T14:30:49.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.860+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v814) ==== 225+0+0 (secure 0 0 0) 0x7fa4a407b0b0 con 0x7fa4b4100ff0 2024-03-30T14:30:49.856 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_objects = 100 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:30:49.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.861+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 <== mon.0 v2:172.21.15.102:3300/0 10 ==== osd_map(814..814 src has 266..814) ==== 628+0+0 (secure 0 0 0) 0x7fa4a4059470 con 0x7fa4b4100ff0 2024-03-30T14:30:49.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.861+0000 7fa49bfff640 1 -- 172.21.15.102:0/453336625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=815}) -- 0x7fa49c07a190 con 0x7fa4b4100ff0 2024-03-30T14:30:49.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.864+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 >> v2:172.21.15.102:6800/622906077 conn(0x7fa49c06db70 msgr2=0x7fa49c070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:49.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.864+0000 7fa4b971e640 1 --2- 172.21.15.102:0/453336625 >> v2:172.21.15.102:6800/622906077 conn(0x7fa49c06db70 0x7fa49c070030 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80095f0 tx=0x7fa4a8005f90 comp rx=0 tx=0).stop 2024-03-30T14:30:49.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.864+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 msgr2=0x7fa4b4198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:49.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.864+0000 7fa4b971e640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b4198aa0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400e9c0 tx=0x7fa4a400ee80 comp rx=0 tx=0).stop 2024-03-30T14:30:49.860 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.865+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 shutdown_connections 2024-03-30T14:30:49.860 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.865+0000 7fa4b971e640 1 --2- 172.21.15.102:0/453336625 >> v2:172.21.15.102:6800/622906077 conn(0x7fa49c06db70 0x7fa49c070030 unknown :-1 s=CLOSED pgs=3479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:49.860 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.865+0000 7fa4b971e640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4b4106640 0x7fa4b41a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.865+0000 7fa4b971e640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b4100ff0 0x7fa4b4198aa0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.865+0000 7fa4b971e640 1 --2- 172.21.15.102:0/453336625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4b4100630 0x7fa4b4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.865+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 >> 172.21.15.102:0/453336625 conn(0x7fa4b4077740 msgr2=0x7fa4b4103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.866+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 shutdown_connections 2024-03-30T14:30:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:49.866+0000 7fa4b971e640 1 -- 172.21.15.102:0/453336625 wait complete. 2024-03-30T14:30:49.872 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 151644 2024-03-30T14:30:50.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]': finished 2024-03-30T14:30:50.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: osdmap e813: 8 total, 8 up, 8 in 2024-03-30T14:30:50.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: pgmap v1746: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:30:50.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]: dispatch 2024-03-30T14:30:50.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:50.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:30:50.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:49 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]': finished 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: osdmap e813: 8 total, 8 up, 8 in 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: pgmap v1746: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]: dispatch 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]': finished 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: osdmap e813: 8 total, 8 up, 8 in 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: pgmap v1746: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]: dispatch 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:30:50.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:49 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:30:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:50 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]': finished 2024-03-30T14:30:51.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:50 smithi138 ceph-mon[33397]: osdmap e814: 8 total, 8 up, 8 in 2024-03-30T14:30:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:50 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]': finished 2024-03-30T14:30:51.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:50 smithi102 ceph-mon[29392]: osdmap e814: 8 total, 8 up, 8 in 2024-03-30T14:30:51.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:50 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/453336625' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "100"}]': finished 2024-03-30T14:30:51.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:50 smithi102 ceph-mon[37717]: osdmap e814: 8 total, 8 up, 8 in 2024-03-30T14:30:51.913 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -ne 0 ']' 2024-03-30T14:30:51.913 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-03-30T14:30:51.913 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put twomore /etc/passwd 2024-03-30T14:30:51.947 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota b802dd6e-da88-4515-b4f3-9f10759279d2 max_bytes 100 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/307958862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d81055f0 msgr2=0x7fd7d81059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/307958862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d81055f0 0x7fd7d81059d0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009900 tx=0x7fd7c802bc30 comp rx=0 tx=0).stop 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/307958862 shutdown_connections 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/307958862 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd7d8101770 0x7fd7d810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/307958862 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7d8100db0 0x7fd7d8101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/307958862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d81055f0 0x7fd7d81059d0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/307958862 >> 172.21.15.102:0/307958862 conn(0x7fd7d8076b60 msgr2=0x7fd7d80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/307958862 shutdown_connections 2024-03-30T14:30:52.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.037+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/307958862 wait complete. 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 Processor -- start 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 -- start start 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d8100db0 0x7fd7d8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd7d8101770 0x7fd7d8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7d81055f0 0x7fd7d81a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d8110100 con 0x7fd7d8100db0 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d810ff80 con 0x7fd7d81055f0 2024-03-30T14:30:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7dcbeb640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d8110280 con 0x7fd7d8101770 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d8100db0 0x7fd7d8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7cffff640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd7d8101770 0x7fd7d8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d8100db0 0x7fd7d8198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44434/0 (socket says 172.21.15.102:44434) 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 -- 172.21.15.102:0/1472302201 learned_addr learned my addr 172.21.15.102:0/1472302201 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d6d9d640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7d81055f0 0x7fd7d81a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 -- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd7d8101770 msgr2=0x7fd7d8198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd7d8101770 0x7fd7d8198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 -- 172.21.15.102:0/1472302201 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7d81055f0 msgr2=0x7fd7d81a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7d81055f0 0x7fd7d81a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7d659c640 1 -- 172.21.15.102:0/1472302201 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d81a3b20 con 0x7fd7d8100db0 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.038+0000 7fd7cffff640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd7d8101770 0x7fd7d8198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.039+0000 7fd7d659c640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d8100db0 0x7fd7d8198560 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8005940 tx=0x7fd7c802c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.039+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8043020 con 0x7fd7d8100db0 2024-03-30T14:30:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.039+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fd7c80043d0 con 0x7fd7d8100db0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.039+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d81a3cf0 con 0x7fd7d8100db0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.039+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8003a90 con 0x7fd7d8100db0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.039+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d81a4150 con 0x7fd7d8100db0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.040+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7d8101dc0 con 0x7fd7d8100db0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.040+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fd7c8046020 con 0x7fd7d8100db0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.041+0000 7fd7cf7fe640 1 --2- 172.21.15.102:0/1472302201 >> v2:172.21.15.102:6800/622906077 conn(0x7fd7c406db70 0x7fd7c4070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.041+0000 7fd7cffff640 1 --2- 172.21.15.102:0/1472302201 >> v2:172.21.15.102:6800/622906077 conn(0x7fd7c406db70 0x7fd7c4070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.041+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(815..815 src has 266..815) ==== 7806+0+0 (secure 0 0 0) 0x7fd7c80afd10 con 0x7fd7d8100db0 2024-03-30T14:30:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.041+0000 7fd7cffff640 1 --2- 172.21.15.102:0/1472302201 >> v2:172.21.15.102:6800/622906077 conn(0x7fd7c406db70 0x7fd7c4070030 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7fd7d81007a0 tx=0x7fd7c000b390 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:30:52.041 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.046+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd7c804d070 con 0x7fd7d8100db0 2024-03-30T14:30:52.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:51 smithi138 ceph-mon[33397]: pgmap v1748: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:52.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:51 smithi102 ceph-mon[29392]: pgmap v1748: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:52.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:51 smithi102 ceph-mon[37717]: pgmap v1748: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:30:52.216 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.220+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"} v 0) -- 0x7fd7d8108b40 con 0x7fd7d8100db0 2024-03-30T14:30:52.897 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:52.901+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v816) ==== 221+0+0 (secure 0 0 0) 0x7fd7c807d1f0 con 0x7fd7d8100db0 2024-03-30T14:30:53.008 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.012+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"} v 0) -- 0x7fd7d81085b0 con 0x7fd7d8100db0 2024-03-30T14:30:53.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:52 smithi138 ceph-mon[33397]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' no longer out of quota; removing NO_QUOTA flag 2024-03-30T14:30:53.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:52 smithi138 ceph-mon[33397]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:30:53.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:52 smithi138 ceph-mon[33397]: osdmap e815: 8 total, 8 up, 8 in 2024-03-30T14:30:53.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:52 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-30T14:30:53.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[29392]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' no longer out of quota; removing NO_QUOTA flag 2024-03-30T14:30:53.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[29392]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:30:53.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[29392]: osdmap e815: 8 total, 8 up, 8 in 2024-03-30T14:30:53.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-30T14:30:53.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[37717]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' no longer out of quota; removing NO_QUOTA flag 2024-03-30T14:30:53.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[37717]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:30:53.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[37717]: osdmap e815: 8 total, 8 up, 8 in 2024-03-30T14:30:53.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:52 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-30T14:30:53.898 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.903+0000 7fd7cf7fe640 1 -- 172.21.15.102:0/1472302201 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v817) ==== 221+0+0 (secure 0 0 0) 0x7fd7c8080c40 con 0x7fd7d8100db0 2024-03-30T14:30:53.898 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_bytes = 100 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:30:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 >> v2:172.21.15.102:6800/622906077 conn(0x7fd7c406db70 msgr2=0x7fd7c4070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/1472302201 >> v2:172.21.15.102:6800/622906077 conn(0x7fd7c406db70 0x7fd7c4070030 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7fd7d81007a0 tx=0x7fd7c000b390 comp rx=0 tx=0).stop 2024-03-30T14:30:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d8100db0 msgr2=0x7fd7d8198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d8100db0 0x7fd7d8198560 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8005940 tx=0x7fd7c802c7d0 comp rx=0 tx=0).stop 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 shutdown_connections 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/1472302201 >> v2:172.21.15.102:6800/622906077 conn(0x7fd7c406db70 0x7fd7c4070030 unknown :-1 s=CLOSED pgs=3481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7d81055f0 0x7fd7d81a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd7d8101770 0x7fd7d8198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 --2- 172.21.15.102:0/1472302201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d8100db0 0x7fd7d8198560 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 >> 172.21.15.102:0/1472302201 conn(0x7fd7d8076b60 msgr2=0x7fd7d8103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 shutdown_connections 2024-03-30T14:30:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:30:53.907+0000 7fd7dcbeb640 1 -- 172.21.15.102:0/1472302201 wait complete. 2024-03-30T14:30:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:+ sleep 30 2024-03-30T14:30:54.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:53 smithi138 ceph-mon[33397]: pgmap v1750: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:54.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]': finished 2024-03-30T14:30:54.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:53 smithi138 ceph-mon[33397]: osdmap e816: 8 total, 8 up, 8 in 2024-03-30T14:30:54.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:53 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-30T14:30:54.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[37717]: pgmap v1750: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:54.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]': finished 2024-03-30T14:30:54.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[37717]: osdmap e816: 8 total, 8 up, 8 in 2024-03-30T14:30:54.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-30T14:30:54.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[29392]: pgmap v1750: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:30:54.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]': finished 2024-03-30T14:30:54.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[29392]: osdmap e816: 8 total, 8 up, 8 in 2024-03-30T14:30:54.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:53 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-30T14:30:55.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:54 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]': finished 2024-03-30T14:30:55.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:54 smithi138 ceph-mon[33397]: osdmap e817: 8 total, 8 up, 8 in 2024-03-30T14:30:55.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:54 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]': finished 2024-03-30T14:30:55.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:54 smithi102 ceph-mon[37717]: osdmap e817: 8 total, 8 up, 8 in 2024-03-30T14:30:55.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:54 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1472302201' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "100"}]': finished 2024-03-30T14:30:55.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:54 smithi102 ceph-mon[29392]: osdmap e817: 8 total, 8 up, 8 in 2024-03-30T14:30:56.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:55 smithi138 ceph-mon[33397]: pgmap v1753: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-30T14:30:56.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:55 smithi102 ceph-mon[29392]: pgmap v1753: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-30T14:30:56.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:55 smithi102 ceph-mon[37717]: pgmap v1753: 176 pgs: 176 active+clean; 698 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-30T14:30:56.916 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:30:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:30:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:30:57.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:56 smithi138 ceph-mon[33397]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' is full (reached quota's max_bytes: 100 B) 2024-03-30T14:30:57.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:56 smithi138 ceph-mon[33397]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:30:57.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:56 smithi138 ceph-mon[33397]: osdmap e818: 8 total, 8 up, 8 in 2024-03-30T14:30:57.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:30:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:30:57.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:56 smithi102 ceph-mon[37717]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' is full (reached quota's max_bytes: 100 B) 2024-03-30T14:30:57.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:56 smithi102 ceph-mon[37717]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:30:57.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:56 smithi102 ceph-mon[37717]: osdmap e818: 8 total, 8 up, 8 in 2024-03-30T14:30:57.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:56 smithi102 ceph-mon[29392]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' is full (reached quota's max_bytes: 100 B) 2024-03-30T14:30:57.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:56 smithi102 ceph-mon[29392]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:30:57.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:56 smithi102 ceph-mon[29392]: osdmap e818: 8 total, 8 up, 8 in 2024-03-30T14:30:58.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:57 smithi138 ceph-mon[33397]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-30T14:30:58.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:57 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:58.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:57 smithi102 ceph-mon[29392]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-30T14:30:58.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:57 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:58.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:57 smithi102 ceph-mon[37717]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-30T14:30:58.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:57 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:30:59.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:59.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:30:59.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:00.186 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:30:59 smithi138 ceph-mon[33397]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-30T14:31:00.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:30:59 smithi102 ceph-mon[29392]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-30T14:31:00.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:30:59 smithi102 ceph-mon[37717]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-30T14:31:02.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:01 smithi138 ceph-mon[33397]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-30T14:31:02.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:01 smithi102 ceph-mon[29392]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-30T14:31:02.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:01 smithi102 ceph-mon[37717]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-30T14:31:04.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:03 smithi138 ceph-mon[33397]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-30T14:31:04.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:03 smithi102 ceph-mon[29392]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-30T14:31:04.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:03 smithi102 ceph-mon[37717]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-30T14:31:06.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:05 smithi102 ceph-mon[29392]: pgmap v1759: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-30T14:31:06.189 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:05 smithi102 ceph-mon[37717]: pgmap v1759: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-30T14:31:06.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:05 smithi138 ceph-mon[33397]: pgmap v1759: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-30T14:31:06.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:31:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:31:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:31:07.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:31:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:31:08.184 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:07 smithi138 ceph-mon[33397]: pgmap v1760: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:31:08.185 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:07 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:07 smithi102 ceph-mon[37717]: pgmap v1760: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:31:08.188 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:07 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:08.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:07 smithi102 ceph-mon[29392]: pgmap v1760: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:31:08.188 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:07 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:10.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:09 smithi138 ceph-mon[33397]: pgmap v1761: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-30T14:31:10.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:09 smithi102 ceph-mon[29392]: pgmap v1761: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-30T14:31:10.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:09 smithi102 ceph-mon[37717]: pgmap v1761: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-03-30T14:31:12.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:11 smithi138 ceph-mon[33397]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:12.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:11 smithi102 ceph-mon[29392]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:12.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:11 smithi102 ceph-mon[37717]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:14.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:13 smithi138 ceph-mon[33397]: pgmap v1763: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:14.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:13 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:14.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:13 smithi102 ceph-mon[29392]: pgmap v1763: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:14.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:13 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:14.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:13 smithi102 ceph-mon[37717]: pgmap v1763: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:14.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:13 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:16.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:15 smithi138 ceph-mon[33397]: pgmap v1764: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:16.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:15 smithi102 ceph-mon[29392]: pgmap v1764: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:16.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:15 smithi102 ceph-mon[37717]: pgmap v1764: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:16.938 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:31:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:31:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:31:17.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:31:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:31:18.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:17 smithi138 ceph-mon[33397]: pgmap v1765: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:18.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:17 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:18.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:17 smithi102 ceph-mon[29392]: pgmap v1765: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:18.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:17 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:18.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:17 smithi102 ceph-mon[37717]: pgmap v1765: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:18.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:17 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:20.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:19 smithi138 ceph-mon[33397]: pgmap v1766: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:20.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:19 smithi102 ceph-mon[29392]: pgmap v1766: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:20.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:19 smithi102 ceph-mon[37717]: pgmap v1766: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:22.437 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:21 smithi138 ceph-mon[33397]: pgmap v1767: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:22.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:21 smithi102 ceph-mon[37717]: pgmap v1767: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:22.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:21 smithi102 ceph-mon[29392]: pgmap v1767: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:23.919 INFO:tasks.workunit.client.0.smithi102.stderr:+ pid=151878 2024-03-30T14:31:23.919 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota b802dd6e-da88-4515-b4f3-9f10759279d2 max_bytes 0 2024-03-30T14:31:23.919 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put two /etc/passwd 2024-03-30T14:31:24.015 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 -- 172.21.15.102:0/748248507 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e01009d0 msgr2=0x7f03e010e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:24.015 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 --2- 172.21.15.102:0/748248507 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e01009d0 0x7f03e010e5d0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009cd0 tx=0x7f03d802c180 comp rx=0 tx=0).stop 2024-03-30T14:31:24.015 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 -- 172.21.15.102:0/748248507 shutdown_connections 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 --2- 172.21.15.102:0/748248507 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e01009d0 0x7f03e010e5d0 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 --2- 172.21.15.102:0/748248507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e0100010 0x7f03e0100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 --2- 172.21.15.102:0/748248507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03e0107950 0x7f03e0107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 -- 172.21.15.102:0/748248507 >> 172.21.15.102:0/748248507 conn(0x7f03e0075970 msgr2=0x7f03e0075d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 -- 172.21.15.102:0/748248507 shutdown_connections 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.016+0000 7f03e5d85640 1 -- 172.21.15.102:0/748248507 wait complete. 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 Processor -- start 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 -- start start 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e0100010 0x7f03e01976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e01009d0 0x7f03e0197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03e0107950 0x7f03e019bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:24.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03e01112c0 con 0x7f03e01009d0 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f03e0111140 con 0x7f03e0107950 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03e5d85640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f03e0111440 con 0x7f03e0100010 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e0100010 0x7f03e01976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03deffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e01009d0 0x7f03e0197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03deffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e01009d0 0x7f03e0197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37566/0 (socket says 172.21.15.102:37566) 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03deffd640 1 -- 172.21.15.102:0/1992131766 learned_addr learned my addr 172.21.15.102:0/1992131766 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e0100010 0x7f03e01976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:59838/0 (socket says 172.21.15.102:59838) 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03dffff640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03e0107950 0x7f03e019bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03deffd640 1 -- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e0100010 msgr2=0x7f03e01976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.017+0000 7f03deffd640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e0100010 0x7f03e01976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03deffd640 1 -- 172.21.15.102:0/1992131766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03e0107950 msgr2=0x7f03e019bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03deffd640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03e0107950 0x7f03e019bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03deffd640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e019c6b0 con 0x7f03e01009d0 2024-03-30T14:31:24.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03dffff640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03e0107950 0x7f03e019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T14:31:24.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03df7fe640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e0100010 0x7f03e01976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:31:24.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03deffd640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e01009d0 0x7f03e0197c00 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f03d400ea60 tx=0x7f03d400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:24.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d400cd20 con 0x7f03e01009d0 2024-03-30T14:31:24.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e019c9a0 con 0x7f03e01009d0 2024-03-30T14:31:24.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f03d40035f0 con 0x7f03e01009d0 2024-03-30T14:31:24.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4010690 con 0x7f03e01009d0 2024-03-30T14:31:24.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.018+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e01a4230 con 0x7f03e01009d0 2024-03-30T14:31:24.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.019+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e0068a80 con 0x7f03e01009d0 2024-03-30T14:31:24.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.019+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f03d4003fb0 con 0x7f03e01009d0 2024-03-30T14:31:24.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.020+0000 7f03dcff9640 1 --2- 172.21.15.102:0/1992131766 >> v2:172.21.15.102:6800/622906077 conn(0x7f03c806db70 0x7f03c8070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:24.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.020+0000 7f03df7fe640 1 --2- 172.21.15.102:0/1992131766 >> v2:172.21.15.102:6800/622906077 conn(0x7f03c806db70 0x7f03c8070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:24.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.020+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(818..818 src has 266..818) ==== 7806+0+0 (secure 0 0 0) 0x7f03d408e4e0 con 0x7f03e01009d0 2024-03-30T14:31:24.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.020+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=819}) -- 0x7f03c8079060 con 0x7f03e01009d0 2024-03-30T14:31:24.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.020+0000 7f03df7fe640 1 --2- 172.21.15.102:0/1992131766 >> v2:172.21.15.102:6800/622906077 conn(0x7f03c806db70 0x7f03c8070030 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7f03d000af30 tx=0x7f03d000a2d0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:24.021 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.025+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f03d405b9c0 con 0x7f03e01009d0 2024-03-30T14:31:24.198 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.198+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"} v 0) -- 0x7f03e0109c00 con 0x7f03e01009d0 2024-03-30T14:31:24.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:23 smithi138 ceph-mon[33397]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:24.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:23 smithi102 ceph-mon[29392]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:24.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:23 smithi102 ceph-mon[37717]: pgmap v1768: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:24.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.969+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v819) ==== 217+0+0 (secure 0 0 0) 0x7f03d405f410 con 0x7f03e01009d0 2024-03-30T14:31:24.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.970+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 8 ==== osd_map(819..819 src has 266..819) ==== 628+0+0 (secure 0 0 0) 0x7f03d408c2f0 con 0x7f03e01009d0 2024-03-30T14:31:24.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:24.970+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=820}) -- 0x7f03c8079fc0 con 0x7f03e01009d0 2024-03-30T14:31:25.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.081+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"} v 0) -- 0x7f03e0109670 con 0x7f03e01009d0 2024-03-30T14:31:25.434 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:24 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:31:25.438 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:24 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:31:25.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:24 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:31:25.968 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.971+0000 7f03dcff9640 1 -- 172.21.15.102:0/1992131766 <== mon.0 v2:172.21.15.102:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v820) ==== 217+0+0 (secure 0 0 0) 0x7f03d4063c40 con 0x7f03e01009d0 2024-03-30T14:31:25.968 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_bytes = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 >> v2:172.21.15.102:6800/622906077 conn(0x7f03c806db70 msgr2=0x7f03c8070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 --2- 172.21.15.102:0/1992131766 >> v2:172.21.15.102:6800/622906077 conn(0x7f03c806db70 0x7f03c8070030 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7f03d000af30 tx=0x7f03d000a2d0 comp rx=0 tx=0).stop 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e01009d0 msgr2=0x7f03e0197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e01009d0 0x7f03e0197c00 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f03d400ea60 tx=0x7f03d400ee60 comp rx=0 tx=0).stop 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 shutdown_connections 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 --2- 172.21.15.102:0/1992131766 >> v2:172.21.15.102:6800/622906077 conn(0x7f03c806db70 0x7f03c8070030 unknown :-1 s=CLOSED pgs=3483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03e0107950 0x7f03e019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e01009d0 0x7f03e0197c00 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 --2- 172.21.15.102:0/1992131766 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f03e0100010 0x7f03e01976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 >> 172.21.15.102:0/1992131766 conn(0x7f03e0075970 msgr2=0x7f03e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:25.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 shutdown_connections 2024-03-30T14:31:25.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:25.976+0000 7f03e5d85640 1 -- 172.21.15.102:0/1992131766 wait complete. 2024-03-30T14:31:25.987 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota b802dd6e-da88-4515-b4f3-9f10759279d2 max_objects 0 2024-03-30T14:31:26.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.077+0000 7f72e2b48640 1 -- 172.21.15.102:0/3644037026 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc101710 msgr2=0x7f72dc101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:26.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.077+0000 7f72e2b48640 1 --2- 172.21.15.102:0/3644037026 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc101710 0x7f72dc101b90 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009890 tx=0x7f72d002bb70 comp rx=0 tx=0).stop 2024-03-30T14:31:26.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- 172.21.15.102:0/3644037026 shutdown_connections 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 --2- 172.21.15.102:0/3644037026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc1020d0 0x7f72dc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 --2- 172.21.15.102:0/3644037026 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc101710 0x7f72dc101b90 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 --2- 172.21.15.102:0/3644037026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72dc107720 0x7f72dc107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- 172.21.15.102:0/3644037026 >> 172.21.15.102:0/3644037026 conn(0x7f72dc0fd1e0 msgr2=0x7f72dc0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- 172.21.15.102:0/3644037026 shutdown_connections 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- 172.21.15.102:0/3644037026 wait complete. 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 Processor -- start 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- start start 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc101710 0x7f72dc19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72dc1020d0 0x7f72dc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc107720 0x7f72dc1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc111240 con 0x7f72dc101710 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1110c0 con 0x7f72dc1020d0 2024-03-30T14:31:26.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e2b48640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f72dc1113c0 con 0x7f72dc107720 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e08e4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc101710 0x7f72dc19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e10e5640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc107720 0x7f72dc1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e08e4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc101710 0x7f72dc19d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37582/0 (socket says 172.21.15.102:37582) 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e08e4640 1 -- 172.21.15.102:0/812110679 learned_addr learned my addr 172.21.15.102:0/812110679 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.078+0000 7f72e10e5640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc107720 0x7f72dc1975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:59854/0 (socket says 172.21.15.102:59854) 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72dbfff640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72dc1020d0 0x7f72dc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e10e5640 1 -- 172.21.15.102:0/812110679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72dc1020d0 msgr2=0x7f72dc19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e10e5640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72dc1020d0 0x7f72dc19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e10e5640 1 -- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc101710 msgr2=0x7f72dc19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e10e5640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc101710 0x7f72dc19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e10e5640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72dc197d20 con 0x7f72dc107720 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e08e4640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc101710 0x7f72dc19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e10e5640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc107720 0x7f72dc1975a0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f72cc005fd0 tx=0x7f72cc00ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc00ccc0 con 0x7f72dc107720 2024-03-30T14:31:26.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f72cc0035f0 con 0x7f72dc107720 2024-03-30T14:31:26.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72dc198010 con 0x7f72dc107720 2024-03-30T14:31:26.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc00ce60 con 0x7f72dc107720 2024-03-30T14:31:26.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.079+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72dc1a42b0 con 0x7f72dc107720 2024-03-30T14:31:26.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.080+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72dc068630 con 0x7f72dc107720 2024-03-30T14:31:26.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.081+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f72cc004000 con 0x7f72dc107720 2024-03-30T14:31:26.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.081+0000 7f72d9ffb640 1 --2- 172.21.15.102:0/812110679 >> v2:172.21.15.102:6800/622906077 conn(0x7f72c406db70 0x7f72c4070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:26.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.081+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(820..820 src has 266..820) ==== 7806+0+0 (secure 0 0 0) 0x7f72cc08e3d0 con 0x7f72dc107720 2024-03-30T14:31:26.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.081+0000 7f72e08e4640 1 --2- 172.21.15.102:0/812110679 >> v2:172.21.15.102:6800/622906077 conn(0x7f72c406db70 0x7f72c4070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:26.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.081+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=821}) -- 0x7f72c4079060 con 0x7f72dc107720 2024-03-30T14:31:26.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.082+0000 7f72e08e4640 1 --2- 172.21.15.102:0/812110679 >> v2:172.21.15.102:6800/622906077 conn(0x7f72c406db70 0x7f72c4070030 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f72d400e740 tx=0x7f72d4009800 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:26.082 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.086+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f72cc009510 con 0x7f72dc107720 2024-03-30T14:31:26.257 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.260+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"} v 0) -- 0x7f72dc109c80 con 0x7f72dc107720 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[29392]: pgmap v1769: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[29392]: osdmap e819: 8 total, 8 up, 8 in 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[37717]: pgmap v1769: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[37717]: osdmap e819: 8 total, 8 up, 8 in 2024-03-30T14:31:26.340 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:25 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:31:26.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:25 smithi138 ceph-mon[33397]: pgmap v1769: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:26.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:31:26.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:25 smithi138 ceph-mon[33397]: osdmap e819: 8 total, 8 up, 8 in 2024-03-30T14:31:26.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:25 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:31:26.688 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:31:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:31:26.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.906+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 7 ==== osd_map(821..821 src has 266..821) ==== 628+0+0 (secure 0 0 0) 0x7f72cc08c1e0 con 0x7f72dc107720 2024-03-30T14:31:26.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.906+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=822}) -- 0x7f72c4079b60 con 0x7f72dc107720 2024-03-30T14:31:26.907 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:26.911+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v821) ==== 221+0+0 (secure 0 0 0) 0x7f72cc05f300 con 0x7f72dc107720 2024-03-30T14:31:27.020 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:27.023+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"} v 0) -- 0x7f72dc1096f0 con 0x7f72dc107720 2024-03-30T14:31:27.336 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:31:27.336 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: osdmap e820: 8 total, 8 up, 8 in 2024-03-30T14:31:27.336 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/812110679' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:27.337 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:27.337 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' no longer out of quota; removing NO_QUOTA flag 2024-03-30T14:31:27.337 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:31:27.337 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:31:27.337 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:26 smithi138 ceph-mon[33397]: osdmap e821: 8 total, 8 up, 8 in 2024-03-30T14:31:27.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:31:27 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: osdmap e820: 8 total, 8 up, 8 in 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/812110679' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' no longer out of quota; removing NO_QUOTA flag 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[37717]: osdmap e821: 8 total, 8 up, 8 in 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1992131766' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:31:27.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: osdmap e820: 8 total, 8 up, 8 in 2024-03-30T14:31:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/812110679' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' no longer out of quota; removing NO_QUOTA flag 2024-03-30T14:31:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:31:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:31:27.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:26 smithi102 ceph-mon[29392]: osdmap e821: 8 total, 8 up, 8 in 2024-03-30T14:31:27.990 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:27.994+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 9 ==== osd_map(822..822 src has 266..822) ==== 628+0+0 (secure 0 0 0) 0x7f72cc0042f0 con 0x7f72dc107720 2024-03-30T14:31:27.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:27.999+0000 7f72d9ffb640 1 -- 172.21.15.102:0/812110679 <== mon.2 v2:172.21.15.102:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v822) ==== 221+0+0 (secure 0 0 0) 0x7f72cc063b30 con 0x7f72dc107720 2024-03-30T14:31:27.995 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_objects = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:31:27.998 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 >> v2:172.21.15.102:6800/622906077 conn(0x7f72c406db70 msgr2=0x7f72c4070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 --2- 172.21.15.102:0/812110679 >> v2:172.21.15.102:6800/622906077 conn(0x7f72c406db70 0x7f72c4070030 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f72d400e740 tx=0x7f72d4009800 comp rx=0 tx=0).stop 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc107720 msgr2=0x7f72dc1975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc107720 0x7f72dc1975a0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f72cc005fd0 tx=0x7f72cc00ed20 comp rx=0 tx=0).stop 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 shutdown_connections 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 --2- 172.21.15.102:0/812110679 >> v2:172.21.15.102:6800/622906077 conn(0x7f72c406db70 0x7f72c4070030 unknown :-1 s=CLOSED pgs=3484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f72dc107720 0x7f72dc1975a0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72dc1020d0 0x7f72dc19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 --2- 172.21.15.102:0/812110679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72dc101710 0x7f72dc19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 >> 172.21.15.102:0/812110679 conn(0x7f72dc0fd1e0 msgr2=0x7f72dc0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 shutdown_connections 2024-03-30T14:31:27.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.003+0000 7f72e2b48640 1 -- 172.21.15.102:0/812110679 wait complete. 2024-03-30T14:31:28.010 INFO:tasks.workunit.client.0.smithi102.stderr:+ wait 151878 2024-03-30T14:31:28.010 INFO:tasks.workunit.client.0.smithi102.stderr:+ '[' 0 -ne 0 ']' 2024-03-30T14:31:28.010 INFO:tasks.workunit.client.0.smithi102.stderr:+ true 2024-03-30T14:31:28.010 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put three /etc/passwd 2024-03-30T14:31:28.047 INFO:tasks.workunit.client.0.smithi102.stderr:++ uuidgen 2024-03-30T14:31:28.048 INFO:tasks.workunit.client.0.smithi102.stderr:+ pp=d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 2024-03-30T14:31:28.048 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool create d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 12 2024-03-30T14:31:28.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 -- 172.21.15.102:0/2579695628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001029c0 msgr2=0x7f2000102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:28.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 --2- 172.21.15.102:0/2579695628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001029c0 0x7f2000102da0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009860 tx=0x7f1ff002bc00 comp rx=0 tx=0).stop 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 -- 172.21.15.102:0/2579695628 shutdown_connections 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 --2- 172.21.15.102:0/2579695628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f200010bd90 0x7f200010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 --2- 172.21.15.102:0/2579695628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f20001032e0 0x7f200010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 --2- 172.21.15.102:0/2579695628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001029c0 0x7f2000102da0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 -- 172.21.15.102:0/2579695628 >> 172.21.15.102:0/2579695628 conn(0x7f2000077830 msgr2=0x7f2000077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 -- 172.21.15.102:0/2579695628 shutdown_connections 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.135+0000 7f2004e5f640 1 -- 172.21.15.102:0/2579695628 wait complete. 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 Processor -- start 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 -- start start 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20001029c0 0x7f200019d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001032e0 0x7f200019da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f200010bd90 0x7f2000197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2000111230 con 0x7f20001029c0 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f20001110b0 con 0x7f200010bd90 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f2004e5f640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f20001113b0 con 0x7f20001032e0 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f1ffe59c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20001029c0 0x7f200019d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f1ff7fff640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001032e0 0x7f200019da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f1ffe59c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20001029c0 0x7f200019d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37614/0 (socket says 172.21.15.102:37614) 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.136+0000 7f1ffe59c640 1 -- 172.21.15.102:0/4214956408 learned_addr learned my addr 172.21.15.102:0/4214956408 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ffed9d640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f200010bd90 0x7f2000197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff7fff640 1 -- 172.21.15.102:0/4214956408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f200010bd90 msgr2=0x7f2000197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff7fff640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f200010bd90 0x7f2000197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff7fff640 1 -- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20001029c0 msgr2=0x7f200019d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff7fff640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20001029c0 0x7f200019d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff7fff640 1 -- 172.21.15.102:0/4214956408 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2000197f20 con 0x7f20001032e0 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ffe59c640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20001029c0 0x7f200019d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:31:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff7fff640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001032e0 0x7f200019da40 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8007b50 tx=0x7f1fe8010420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe801b020 con 0x7f20001032e0 2024-03-30T14:31:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f1fe800e0d0 con 0x7f20001032e0 2024-03-30T14:31:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20001981b0 con 0x7f20001032e0 2024-03-30T14:31:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8005540 con 0x7f20001032e0 2024-03-30T14:31:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.137+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2000100720 con 0x7f20001032e0 2024-03-30T14:31:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.138+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2000198520 con 0x7f20001032e0 2024-03-30T14:31:28.137 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.138+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f1fe8010b30 con 0x7f20001032e0 2024-03-30T14:31:28.137 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.139+0000 7f1ff77fe640 1 --2- 172.21.15.102:0/4214956408 >> v2:172.21.15.102:6800/622906077 conn(0x7f1fec06db70 0x7f1fec070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:28.137 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.139+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(822..822 src has 266..822) ==== 7806+0+0 (secure 0 0 0) 0x7f1fe8091920 con 0x7f20001032e0 2024-03-30T14:31:28.137 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.139+0000 7f1ffe59c640 1 --2- 172.21.15.102:0/4214956408 >> v2:172.21.15.102:6800/622906077 conn(0x7f1fec06db70 0x7f1fec070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:28.137 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.139+0000 7f1ffe59c640 1 --2- 172.21.15.102:0/4214956408 >> v2:172.21.15.102:6800/622906077 conn(0x7f1fec06db70 0x7f1fec070030 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0000c00 tx=0x7f1ff002e040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:28.139 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.143+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1fe8025020 con 0x7f20001032e0 2024-03-30T14:31:28.315 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:28.320+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12} v 0) -- 0x7f2000068a80 con 0x7f20001032e0 2024-03-30T14:31:28.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:27 smithi138 ceph-mon[33397]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:28.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:27 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/812110679' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:28.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:27 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:28.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:27 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:28.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[37717]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:28.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/812110679' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:28.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:28.438 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:28.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[29392]: pgmap v1772: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:28.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/812110679' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:28.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:31:28.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:27 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:29.325 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.329+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]=0 pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' created v823) ==== 176+0+0 (secure 0 0 0) 0x7f1fe8010e20 con 0x7f20001032e0 2024-03-30T14:31:29.434 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.439+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12} v 0) -- 0x7f200006a670 con 0x7f20001032e0 2024-03-30T14:31:29.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:31:29.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:28 smithi138 ceph-mon[33397]: osdmap e822: 8 total, 8 up, 8 in 2024-03-30T14:31:29.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:28 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:31:29.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:28 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4214956408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:29.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:28 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:29.435 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:29.436 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.440+0000 7f1ff77fe640 1 -- 172.21.15.102:0/4214956408 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]=0 pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' already exists v823) ==== 183+0+0 (secure 0 0 0) 0x7f1fe805ee00 con 0x7f20001032e0 2024-03-30T14:31:29.436 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' already exists 2024-03-30T14:31:29.437 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 >> v2:172.21.15.102:6800/622906077 conn(0x7f1fec06db70 msgr2=0x7f1fec070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:29.437 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 --2- 172.21.15.102:0/4214956408 >> v2:172.21.15.102:6800/622906077 conn(0x7f1fec06db70 0x7f1fec070030 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0000c00 tx=0x7f1ff002e040 comp rx=0 tx=0).stop 2024-03-30T14:31:29.437 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001032e0 msgr2=0x7f200019da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001032e0 0x7f200019da40 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8007b50 tx=0x7f1fe8010420 comp rx=0 tx=0).stop 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 shutdown_connections 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 --2- 172.21.15.102:0/4214956408 >> v2:172.21.15.102:6800/622906077 conn(0x7f1fec06db70 0x7f1fec070030 unknown :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f200010bd90 0x7f2000197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20001032e0 0x7f200019da40 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 --2- 172.21.15.102:0/4214956408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20001029c0 0x7f200019d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.441+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 >> 172.21.15.102:0/4214956408 conn(0x7f2000077830 msgr2=0x7f2000103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.442+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 shutdown_connections 2024-03-30T14:31:29.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.442+0000 7f2004e5f640 1 -- 172.21.15.102:0/4214956408 wait complete. 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[37717]: osdmap e822: 8 total, 8 up, 8 in 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4214956408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:31:29.439 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[29392]: osdmap e822: 8 total, 8 up, 8 in 2024-03-30T14:31:29.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:31:29.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4214956408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:29.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:29.440 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:29.446 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool application enable d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 rados 2024-03-30T14:31:29.530 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 -- 172.21.15.102:0/3861153714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd31410bd00 msgr2=0x7fd31410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 --2- 172.21.15.102:0/3861153714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd31410bd00 0x7fd31410e1a0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009be0 tx=0x7fd30c02c010 comp rx=0 tx=0).stop 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 -- 172.21.15.102:0/3861153714 shutdown_connections 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 --2- 172.21.15.102:0/3861153714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd31410bd00 0x7fd31410e1a0 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 --2- 172.21.15.102:0/3861153714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3141032e0 0x7fd31410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 --2- 172.21.15.102:0/3861153714 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3141029c0 0x7fd314102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 -- 172.21.15.102:0/3861153714 >> 172.21.15.102:0/3861153714 conn(0x7fd314077770 msgr2=0x7fd314077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 -- 172.21.15.102:0/3861153714 shutdown_connections 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.532+0000 7fd31a22e640 1 -- 172.21.15.102:0/3861153714 wait complete. 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 Processor -- start 2024-03-30T14:31:29.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 -- start start 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3141029c0 0x7fd3141976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3141032e0 0x7fd314197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd31410bd00 0x7fd31419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd314111210 con 0x7fd31410bd00 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd314111090 con 0x7fd3141032e0 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd31a22e640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd314111390 con 0x7fd3141029c0 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.533+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3141029c0 0x7fd3141976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd313fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd31410bd00 0x7fd31419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd3137fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3141029c0 0x7fd3141976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:59930/0 (socket says 172.21.15.102:59930) 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd3137fe640 1 -- 172.21.15.102:0/2242521206 learned_addr learned my addr 172.21.15.102:0/2242521206 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd30affd640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3141032e0 0x7fd314197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:29.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd313fff640 1 -- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3141029c0 msgr2=0x7fd3141976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd313fff640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3141029c0 0x7fd3141976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd313fff640 1 -- 172.21.15.102:0/2242521206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3141032e0 msgr2=0x7fd314197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd313fff640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3141032e0 0x7fd314197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd313fff640 1 -- 172.21.15.102:0/2242521206 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31419c6d0 con 0x7fd31410bd00 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd3137fe640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3141029c0 0x7fd3141976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd313fff640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd31410bd00 0x7fd31419bfa0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fd30c02c550 tx=0x7fd30c033d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c043020 con 0x7fd31410bd00 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31419c960 con 0x7fd31410bd00 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fd30c003b80 con 0x7fd31410bd00 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3141a4250 con 0x7fd31410bd00 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.534+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c03e650 con 0x7fd31410bd00 2024-03-30T14:31:29.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.535+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd314068630 con 0x7fd31410bd00 2024-03-30T14:31:29.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.536+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fd30c046020 con 0x7fd31410bd00 2024-03-30T14:31:29.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.536+0000 7fd3117fa640 1 --2- 172.21.15.102:0/2242521206 >> v2:172.21.15.102:6800/622906077 conn(0x7fd30006db70 0x7fd300070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:29.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.536+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(823..823 src has 266..823) ==== 8181+0+0 (secure 0 0 0) 0x7fd30c0b0a10 con 0x7fd31410bd00 2024-03-30T14:31:29.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.536+0000 7fd3137fe640 1 --2- 172.21.15.102:0/2242521206 >> v2:172.21.15.102:6800/622906077 conn(0x7fd30006db70 0x7fd300070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:29.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.536+0000 7fd3137fe640 1 --2- 172.21.15.102:0/2242521206 >> v2:172.21.15.102:6800/622906077 conn(0x7fd30006db70 0x7fd300070030 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7fd3040104e0 tx=0x7fd30400b410 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:29.536 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.541+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd30c04d020 con 0x7fd31410bd00 2024-03-30T14:31:29.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:29.713+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"} v 0) -- 0x7fd314106590 con 0x7fd31410bd00 2024-03-30T14:31:30.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:30.325+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]=0 enabled application 'rados' on pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' v824) ==== 213+0+0 (secure 0 0 0) 0x7fd30c081770 con 0x7fd31410bd00 2024-03-30T14:31:30.434 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:30.439+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"} v 0) -- 0x7fd3141061a0 con 0x7fd31410bd00 2024-03-30T14:31:30.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:30 smithi138 ceph-mon[33397]: pgmap v1775: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:30.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]': finished 2024-03-30T14:31:30.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:30 smithi138 ceph-mon[33397]: osdmap e823: 8 total, 8 up, 8 in 2024-03-30T14:31:30.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/4214956408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:30.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:30 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:30.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:30 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]: dispatch 2024-03-30T14:31:30.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[29392]: pgmap v1775: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:30.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]': finished 2024-03-30T14:31:30.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[29392]: osdmap e823: 8 total, 8 up, 8 in 2024-03-30T14:31:30.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/4214956408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:30.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:30.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]: dispatch 2024-03-30T14:31:30.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[37717]: pgmap v1775: 176 pgs: 176 active+clean; 721 KiB data, 853 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:30.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]': finished 2024-03-30T14:31:30.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[37717]: osdmap e823: 8 total, 8 up, 8 in 2024-03-30T14:31:30.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/4214956408' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:30.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pg_num": 12}]: dispatch 2024-03-30T14:31:30.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:30 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]: dispatch 2024-03-30T14:31:31.336 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.340+0000 7fd3117fa640 1 -- 172.21.15.102:0/2242521206 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]=0 enabled application 'rados' on pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' v825) ==== 213+0+0 (secure 0 0 0) 0x7fd30c085fa0 con 0x7fd31410bd00 2024-03-30T14:31:31.336 INFO:tasks.workunit.client.0.smithi102.stderr:enabled application 'rados' on pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' 2024-03-30T14:31:31.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 >> v2:172.21.15.102:6800/622906077 conn(0x7fd30006db70 msgr2=0x7fd300070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:31.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 --2- 172.21.15.102:0/2242521206 >> v2:172.21.15.102:6800/622906077 conn(0x7fd30006db70 0x7fd300070030 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7fd3040104e0 tx=0x7fd30400b410 comp rx=0 tx=0).stop 2024-03-30T14:31:31.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd31410bd00 msgr2=0x7fd31419bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:31.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd31410bd00 0x7fd31419bfa0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fd30c02c550 tx=0x7fd30c033d10 comp rx=0 tx=0).stop 2024-03-30T14:31:31.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 shutdown_connections 2024-03-30T14:31:31.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 --2- 172.21.15.102:0/2242521206 >> v2:172.21.15.102:6800/622906077 conn(0x7fd30006db70 0x7fd300070030 unknown :-1 s=CLOSED pgs=3487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd31410bd00 0x7fd31419bfa0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3141032e0 0x7fd314197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 --2- 172.21.15.102:0/2242521206 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3141029c0 0x7fd3141976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.344+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 >> 172.21.15.102:0/2242521206 conn(0x7fd314077770 msgr2=0x7fd314103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:31.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.345+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 shutdown_connections 2024-03-30T14:31:31.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.345+0000 7fd31a22e640 1 -- 172.21.15.102:0/2242521206 wait complete. 2024-03-30T14:31:31.353 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 max_objects 10 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 -- 172.21.15.102:0/3594817043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c10bd90 msgr2=0x7fab3c10e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 --2- 172.21.15.102:0/3594817043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c10bd90 0x7fab3c10e1c0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fab34009ca0 tx=0x7fab3402c180 comp rx=0 tx=0).stop 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 -- 172.21.15.102:0/3594817043 shutdown_connections 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 --2- 172.21.15.102:0/3594817043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c10bd90 0x7fab3c10e1c0 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 --2- 172.21.15.102:0/3594817043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1032e0 0x7fab3c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 --2- 172.21.15.102:0/3594817043 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab3c1029c0 0x7fab3c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 -- 172.21.15.102:0/3594817043 >> 172.21.15.102:0/3594817043 conn(0x7fab3c077740 msgr2=0x7fab3c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 -- 172.21.15.102:0/3594817043 shutdown_connections 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.444+0000 7fab411d0640 1 -- 172.21.15.102:0/3594817043 wait complete. 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.445+0000 7fab411d0640 1 Processor -- start 2024-03-30T14:31:31.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.445+0000 7fab411d0640 1 -- start start 2024-03-30T14:31:31.444 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.445+0000 7fab411d0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1029c0 0x7fab3c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:31.444 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab411d0640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c1032e0 0x7fab3c100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:31.444 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab411d0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab3c10bd90 0x7fab3c101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:31.444 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab411d0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c111230 con 0x7fab3c1029c0 2024-03-30T14:31:31.444 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab411d0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c1110b0 con 0x7fab3c10bd90 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab411d0640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fab3c1113b0 con 0x7fab3c1032e0 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3a59c640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c1032e0 0x7fab3c100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1029c0 0x7fab3c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3a59c640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c1032e0 0x7fab3c100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:44436/0 (socket says 172.21.15.102:44436) 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3a59c640 1 -- 172.21.15.102:0/1728057985 learned_addr learned my addr 172.21.15.102:0/1728057985 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1029c0 0x7fab3c100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42404/0 (socket says 172.21.15.102:42404) 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 -- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c1032e0 msgr2=0x7fab3c100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c1032e0 0x7fab3c100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3b59e640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab3c10bd90 0x7fab3c101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 -- 172.21.15.102:0/1728057985 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab3c10bd90 msgr2=0x7fab3c101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab3c10bd90 0x7fab3c101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 -- 172.21.15.102:0/1728057985 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c1a5ad0 con 0x7fab3c1029c0 2024-03-30T14:31:31.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3a59c640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c1032e0 0x7fab3c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab3ad9d640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1029c0 0x7fab3c100610 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c970 tx=0x7fab2c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c007bb0 con 0x7fab3c1029c0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3c1a5dc0 con 0x7fab3c1029c0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7fab2c007d50 con 0x7fab3c1029c0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3c1a6290 con 0x7fab3c1029c0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.446+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0186e0 con 0x7fab3c1029c0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.447+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab3c06a050 con 0x7fab3c1029c0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.448+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7fab2c018940 con 0x7fab3c1029c0 2024-03-30T14:31:31.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.448+0000 7fab23fff640 1 --2- 172.21.15.102:0/1728057985 >> v2:172.21.15.102:6800/622906077 conn(0x7fab2406dba0 0x7fab24070060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:31:31.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.448+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(825..825 src has 266..825) ==== 8194+0+0 (secure 0 0 0) 0x7fab2c08e3a0 con 0x7fab3c1029c0 2024-03-30T14:31:31.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.448+0000 7fab3a59c640 1 --2- 172.21.15.102:0/1728057985 >> v2:172.21.15.102:6800/622906077 conn(0x7fab2406dba0 0x7fab24070060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:31:31.447 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.449+0000 7fab3a59c640 1 --2- 172.21.15.102:0/1728057985 >> v2:172.21.15.102:6800/622906077 conn(0x7fab2406dba0 0x7fab24070060 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7fab300095f0 tx=0x7fab30005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:31:31.448 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.453+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fab2c014030 con 0x7fab3c1029c0 2024-03-30T14:31:31.624 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:31.628+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"} v 0) -- 0x7fab3c1a65b0 con 0x7fab3c1029c0 2024-03-30T14:31:31.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]': finished 2024-03-30T14:31:31.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:31 smithi138 ceph-mon[33397]: osdmap e824: 8 total, 8 up, 8 in 2024-03-30T14:31:31.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:31 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]: dispatch 2024-03-30T14:31:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]': finished 2024-03-30T14:31:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:31 smithi102 ceph-mon[29392]: osdmap e824: 8 total, 8 up, 8 in 2024-03-30T14:31:31.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:31 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]: dispatch 2024-03-30T14:31:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]': finished 2024-03-30T14:31:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:31 smithi102 ceph-mon[37717]: osdmap e824: 8 total, 8 up, 8 in 2024-03-30T14:31:31.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:31 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]: dispatch 2024-03-30T14:31:32.337 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:32.342+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 v826) ==== 223+0+0 (secure 0 0 0) 0x7fab2c05b6a0 con 0x7fab3c1029c0 2024-03-30T14:31:32.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:32.451+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"} v 0) -- 0x7fab3c1a6780 con 0x7fab3c1029c0 2024-03-30T14:31:32.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:32 smithi138 ceph-mon[33397]: pgmap v1778: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-30T14:31:32.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]': finished 2024-03-30T14:31:32.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:32 smithi138 ceph-mon[33397]: osdmap e825: 8 total, 8 up, 8 in 2024-03-30T14:31:32.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:32 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:31:32.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[29392]: pgmap v1778: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-30T14:31:32.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]': finished 2024-03-30T14:31:32.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[29392]: osdmap e825: 8 total, 8 up, 8 in 2024-03-30T14:31:32.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:31:32.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[37717]: pgmap v1778: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-30T14:31:32.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2242521206' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "app": "rados"}]': finished 2024-03-30T14:31:32.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[37717]: osdmap e825: 8 total, 8 up, 8 in 2024-03-30T14:31:32.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:32 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:31:33.351 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.355+0000 7fab23fff640 1 -- 172.21.15.102:0/1728057985 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 v827) ==== 223+0+0 (secure 0 0 0) 0x7fab2c05f0f0 con 0x7fab3c1029c0 2024-03-30T14:31:33.351 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_objects = 10 for pool d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 >> v2:172.21.15.102:6800/622906077 conn(0x7fab2406dba0 msgr2=0x7fab24070060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 --2- 172.21.15.102:0/1728057985 >> v2:172.21.15.102:6800/622906077 conn(0x7fab2406dba0 0x7fab24070060 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7fab300095f0 tx=0x7fab30005f90 comp rx=0 tx=0).stop 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1029c0 msgr2=0x7fab3c100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1029c0 0x7fab3c100610 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c970 tx=0x7fab2c00ce30 comp rx=0 tx=0).stop 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 shutdown_connections 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 --2- 172.21.15.102:0/1728057985 >> v2:172.21.15.102:6800/622906077 conn(0x7fab2406dba0 0x7fab24070060 unknown :-1 s=CLOSED pgs=3488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab3c10bd90 0x7fab3c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab3c1032e0 0x7fab3c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 --2- 172.21.15.102:0/1728057985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1029c0 0x7fab3c100610 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:31:33.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 >> 172.21.15.102:0/1728057985 conn(0x7fab3c077740 msgr2=0x7fab3c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:31:33.356 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 shutdown_connections 2024-03-30T14:31:33.356 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:31:33.359+0000 7fab411d0640 1 -- 172.21.15.102:0/1728057985 wait complete. 2024-03-30T14:31:33.366 INFO:tasks.workunit.client.0.smithi102.stderr:+ sleep 30 2024-03-30T14:31:33.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:31:33.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:33 smithi138 ceph-mon[33397]: osdmap e826: 8 total, 8 up, 8 in 2024-03-30T14:31:33.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:33 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:31:33.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:31:33.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:33 smithi102 ceph-mon[29392]: osdmap e826: 8 total, 8 up, 8 in 2024-03-30T14:31:33.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:33 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:31:33.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:31:33.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:33 smithi102 ceph-mon[37717]: osdmap e826: 8 total, 8 up, 8 in 2024-03-30T14:31:33.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:33 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]: dispatch 2024-03-30T14:31:34.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:34 smithi138 ceph-mon[33397]: pgmap v1781: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-30T14:31:34.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:31:34.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:34 smithi138 ceph-mon[33397]: osdmap e827: 8 total, 8 up, 8 in 2024-03-30T14:31:34.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:34 smithi102 ceph-mon[29392]: pgmap v1781: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-30T14:31:34.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:31:34.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:34 smithi102 ceph-mon[29392]: osdmap e827: 8 total, 8 up, 8 in 2024-03-30T14:31:34.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:34 smithi102 ceph-mon[37717]: pgmap v1781: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-30T14:31:34.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1728057985' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "field": "max_objects", "val": "10"}]': finished 2024-03-30T14:31:34.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:34 smithi102 ceph-mon[37717]: osdmap e827: 8 total, 8 up, 8 in 2024-03-30T14:31:36.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:36 smithi138 ceph-mon[33397]: pgmap v1783: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:31:36.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:36 smithi102 ceph-mon[29392]: pgmap v1783: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:31:36.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:36 smithi102 ceph-mon[37717]: pgmap v1783: 188 pgs: 12 creating+peering, 176 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:31:36.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:31:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:31:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:31:37.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:31:37 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:31:38.684 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:38 smithi138 ceph-mon[33397]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:38.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:38.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:38 smithi102 ceph-mon[29392]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:38.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:38.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:38 smithi102 ceph-mon[37717]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:38.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:40 smithi138 ceph-mon[33397]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 678 B/s rd, 0 op/s 2024-03-30T14:31:40.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:40 smithi102 ceph-mon[29392]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 678 B/s rd, 0 op/s 2024-03-30T14:31:40.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:40 smithi102 ceph-mon[37717]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 678 B/s rd, 0 op/s 2024-03-30T14:31:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:42 smithi138 ceph-mon[33397]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:42.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:42 smithi102 ceph-mon[29392]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:42.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:42 smithi102 ceph-mon[37717]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:43.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:43.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:44.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:44 smithi138 ceph-mon[33397]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:31:44.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:44 smithi102 ceph-mon[29392]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:31:44.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:44 smithi102 ceph-mon[37717]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:31:46.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:46 smithi138 ceph-mon[33397]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-30T14:31:46.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:46 smithi102 ceph-mon[29392]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-30T14:31:46.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:46 smithi102 ceph-mon[37717]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-30T14:31:46.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:31:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:31:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:31:47.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:31:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:31:48.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:48 smithi138 ceph-mon[33397]: pgmap v1789: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:48.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:48 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:48.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:48 smithi102 ceph-mon[29392]: pgmap v1789: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:48.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:48 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:48.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:48 smithi102 ceph-mon[37717]: pgmap v1789: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:48.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:48 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:50.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:50 smithi138 ceph-mon[33397]: pgmap v1790: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:31:50.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:50 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:50.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:50 smithi102 ceph-mon[29392]: pgmap v1790: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:31:50.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:50 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:50.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:50 smithi102 ceph-mon[37717]: pgmap v1790: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:31:50.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:50 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:31:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:31:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:31:51.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:31:51.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:31:51.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:31:51.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:31:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:31:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:31:51.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:31:52.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:52 smithi138 ceph-mon[33397]: pgmap v1791: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:52.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:52 smithi102 ceph-mon[29392]: pgmap v1791: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:52.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:52 smithi102 ceph-mon[37717]: pgmap v1791: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:54.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:54 smithi138 ceph-mon[33397]: pgmap v1792: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:31:54.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:54 smithi102 ceph-mon[29392]: pgmap v1792: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:31:54.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:54 smithi102 ceph-mon[37717]: pgmap v1792: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:31:56.741 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:56 smithi102 ceph-mon[29392]: pgmap v1793: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:56.741 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:56 smithi102 ceph-mon[37717]: pgmap v1793: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:56.742 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:31:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:31:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:31:56.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:56 smithi138 ceph-mon[33397]: pgmap v1793: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:31:57.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:31:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:31:58.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:58 smithi138 ceph-mon[33397]: pgmap v1794: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:58.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:58 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:58.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:31:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:58.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:58 smithi102 ceph-mon[29392]: pgmap v1794: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:58.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:58 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:58.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:31:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:31:58.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:58 smithi102 ceph-mon[37717]: pgmap v1794: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:31:58.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:58 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:31:58.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:31:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:00.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:00 smithi138 ceph-mon[33397]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:32:00.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:00 smithi102 ceph-mon[29392]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:32:00.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:00 smithi102 ceph-mon[37717]: pgmap v1795: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:32:02.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:02 smithi138 ceph-mon[33397]: pgmap v1796: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:02.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:02 smithi102 ceph-mon[29392]: pgmap v1796: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:02 smithi102 ceph-mon[37717]: pgmap v1796: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:03.368 INFO:tasks.workunit.client.0.smithi102.stderr:++ seq 1 10 2024-03-30T14:32:03.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.369 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj1 /etc/passwd 2024-03-30T14:32:03.408 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.409 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj2 /etc/passwd 2024-03-30T14:32:03.445 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.446 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj3 /etc/passwd 2024-03-30T14:32:03.483 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.484 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj4 /etc/passwd 2024-03-30T14:32:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj5 /etc/passwd 2024-03-30T14:32:03.553 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.553 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj6 /etc/passwd 2024-03-30T14:32:03.587 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.587 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj7 /etc/passwd 2024-03-30T14:32:03.621 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.621 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj8 /etc/passwd 2024-03-30T14:32:03.656 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.656 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj9 /etc/passwd 2024-03-30T14:32:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:+ for f in `seq 1 10` 2024-03-30T14:32:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 put obj10 /etc/passwd 2024-03-30T14:32:03.724 INFO:tasks.workunit.client.0.smithi102.stderr:+ sleep 30 2024-03-30T14:32:04.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:04 smithi138 ceph-mon[33397]: pgmap v1797: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:04.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:04 smithi102 ceph-mon[29392]: pgmap v1797: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:04.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:04 smithi102 ceph-mon[37717]: pgmap v1797: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:06.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:06 smithi138 ceph-mon[33397]: pgmap v1798: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:06.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:06 smithi102 ceph-mon[29392]: pgmap v1798: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:06.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:06 smithi102 ceph-mon[37717]: pgmap v1798: 188 pgs: 188 active+clean; 744 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:06.939 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:32:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:32:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:32:07.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:32:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:32:08.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:08 smithi138 ceph-mon[33397]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-30T14:32:08.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:08 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:08.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:08 smithi102 ceph-mon[29392]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-30T14:32:08.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:08 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:08.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:08 smithi102 ceph-mon[37717]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-30T14:32:08.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:08 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:10.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:10 smithi138 ceph-mon[33397]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-30T14:32:10.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:10 smithi102 ceph-mon[29392]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-30T14:32:10.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:10 smithi102 ceph-mon[37717]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-30T14:32:11.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:11 smithi138 ceph-mon[33397]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-30T14:32:11.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:11 smithi102 ceph-mon[29392]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-30T14:32:11.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:11 smithi102 ceph-mon[37717]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-30T14:32:12.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:12 smithi138 ceph-mon[33397]: pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' is full (reached quota's max_objects: 10) 2024-03-30T14:32:12.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:12 smithi138 ceph-mon[33397]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:32:12.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:12 smithi138 ceph-mon[33397]: osdmap e828: 8 total, 8 up, 8 in 2024-03-30T14:32:12.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:12 smithi102 ceph-mon[29392]: pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' is full (reached quota's max_objects: 10) 2024-03-30T14:32:12.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:12 smithi102 ceph-mon[29392]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:32:12.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:12 smithi102 ceph-mon[29392]: osdmap e828: 8 total, 8 up, 8 in 2024-03-30T14:32:12.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:12 smithi102 ceph-mon[37717]: pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' is full (reached quota's max_objects: 10) 2024-03-30T14:32:12.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:12 smithi102 ceph-mon[37717]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:32:12.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:12 smithi102 ceph-mon[37717]: osdmap e828: 8 total, 8 up, 8 in 2024-03-30T14:32:14.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:14 smithi138 ceph-mon[33397]: pgmap v1803: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:32:14.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:14 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:14.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:14 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:14.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:14 smithi102 ceph-mon[29392]: pgmap v1803: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:32:14.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:14 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:14.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:14 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:14.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:14 smithi102 ceph-mon[37717]: pgmap v1803: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:32:14.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:14 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:14.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:14 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:16.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:16 smithi138 ceph-mon[33397]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:32:16.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:16 smithi102 ceph-mon[29392]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:32:16.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:32:16 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:32:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:32:16.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:16 smithi102 ceph-mon[37717]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-30T14:32:17.435 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:32:17 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:32:18.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:18 smithi138 ceph-mon[33397]: pgmap v1805: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:18.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:18 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:18.688 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:18 smithi102 ceph-mon[29392]: pgmap v1805: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:18.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:18 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:18.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:18 smithi102 ceph-mon[37717]: pgmap v1805: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:18.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:18 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:20.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:20 smithi138 ceph-mon[33397]: pgmap v1806: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:20.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:20 smithi102 ceph-mon[29392]: pgmap v1806: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:20.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:20 smithi102 ceph-mon[37717]: pgmap v1806: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:22.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:22 smithi138 ceph-mon[33397]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:22.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:22 smithi102 ceph-mon[29392]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:22.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:22 smithi102 ceph-mon[37717]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:24.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:24 smithi138 ceph-mon[33397]: pgmap v1808: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-30T14:32:24.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:24 smithi102 ceph-mon[29392]: pgmap v1808: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-30T14:32:24.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:24 smithi102 ceph-mon[37717]: pgmap v1808: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-03-30T14:32:26.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:26 smithi138 ceph-mon[33397]: pgmap v1809: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:26.758 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:26 smithi102 ceph-mon[29392]: pgmap v1809: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:26.759 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:26 smithi102 ceph-mon[37717]: pgmap v1809: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:26.759 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:32:26 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:32:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:32:27.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:32:27 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:32:28.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:28 smithi138 ceph-mon[33397]: pgmap v1810: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:28.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:28 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:28.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:28 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:28.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:28 smithi102 ceph-mon[29392]: pgmap v1810: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:28.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:28 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:28.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:28 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:28.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:28 smithi102 ceph-mon[37717]: pgmap v1810: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:28.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:28 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:28.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:28 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:30.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:30 smithi138 ceph-mon[33397]: pgmap v1811: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:30.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:30 smithi102 ceph-mon[29392]: pgmap v1811: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:30.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:30 smithi102 ceph-mon[37717]: pgmap v1811: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:32.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:32 smithi138 ceph-mon[33397]: pgmap v1812: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:32.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:32 smithi102 ceph-mon[29392]: pgmap v1812: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:32.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:32 smithi102 ceph-mon[37717]: pgmap v1812: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:33.726 INFO:tasks.workunit.client.0.smithi102.stderr:+ rados -p b802dd6e-da88-4515-b4f3-9f10759279d2 put threemore /etc/passwd 2024-03-30T14:32:33.767 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota b802dd6e-da88-4515-b4f3-9f10759279d2 max_bytes 0 2024-03-30T14:32:33.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 -- 172.21.15.102:0/3241284255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 msgr2=0x7f34fc107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:33.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 --2- 172.21.15.102:0/3241284255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 0x7f34fc107af0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009890 tx=0x7f34ec02bb70 comp rx=0 tx=0).stop 2024-03-30T14:32:33.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 -- 172.21.15.102:0/3241284255 shutdown_connections 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 --2- 172.21.15.102:0/3241284255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc1020c0 0x7f34fc10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 --2- 172.21.15.102:0/3241284255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34fc101700 0x7f34fc101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 --2- 172.21.15.102:0/3241284255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 0x7f34fc107af0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 -- 172.21.15.102:0/3241284255 >> 172.21.15.102:0/3241284255 conn(0x7f34fc0fd210 msgr2=0x7f34fc0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.858+0000 7f3502472640 1 -- 172.21.15.102:0/3241284255 shutdown_connections 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.859+0000 7f3502472640 1 -- 172.21.15.102:0/3241284255 wait complete. 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.859+0000 7f3502472640 1 Processor -- start 2024-03-30T14:32:33.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.859+0000 7f3502472640 1 -- start start 2024-03-30T14:32:33.860 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.859+0000 7f3502472640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc101700 0x7f34fc197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:33.860 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.859+0000 7f3502472640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34fc1020c0 0x7f34fc197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:33.860 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3502472640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 0x7f34fc19c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3502472640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f34fc1111f0 con 0x7f34fc107710 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3502472640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f34fc111070 con 0x7f34fc1020c0 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3502472640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f34fc111370 con 0x7f34fc101700 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc101700 0x7f34fc197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc101700 0x7f34fc197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:33162/0 (socket says 172.21.15.102:33162) 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 -- 172.21.15.102:0/1198612153 learned_addr learned my addr 172.21.15.102:0/1198612153 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3500a0f640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 0x7f34fc19c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fb7fe640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34fc1020c0 0x7f34fc197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 -- 172.21.15.102:0/1198612153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34fc1020c0 msgr2=0x7f34fc197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34fc1020c0 0x7f34fc197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:33.861 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 -- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 msgr2=0x7f34fc19c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 0x7f34fc19c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34fc19c750 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3500a0f640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 0x7f34fc19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34fbfff640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc101700 0x7f34fc197760 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f34ec000c00 tx=0x7f34ec034840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec039040 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34fc19c9e0 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f34ec02c9c0 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec0035f0 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.860+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34fc073980 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.861+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34fc102e40 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.862+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f34ec03d5f0 con 0x7f34fc101700 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.862+0000 7f34f97fa640 1 --2- 172.21.15.102:0/1198612153 >> v2:172.21.15.102:6800/622906077 conn(0x7f34e406db70 0x7f34e4070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.862+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(828..828 src has 266..828) ==== 8194+0+0 (secure 0 0 0) 0x7f34ec0b3be0 con 0x7f34fc101700 2024-03-30T14:32:33.863 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.862+0000 7f34fb7fe640 1 --2- 172.21.15.102:0/1198612153 >> v2:172.21.15.102:6800/622906077 conn(0x7f34e406db70 0x7f34e4070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:33.863 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.862+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=829}) -- 0x7f34e4079880 con 0x7f34fc101700 2024-03-30T14:32:33.863 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.863+0000 7f34fb7fe640 1 --2- 172.21.15.102:0/1198612153 >> v2:172.21.15.102:6800/622906077 conn(0x7f34e406db70 0x7f34e4070030 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0ff930 tx=0x7f34f000b410 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:32:33.864 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:33.867+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f34ec084930 con 0x7f34fc101700 2024-03-30T14:32:34.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:34.041+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"} v 0) -- 0x7f34fc198ab0 con 0x7f34fc101700 2024-03-30T14:32:34.392 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:34.394+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 7 ==== osd_map(829..829 src has 266..829) ==== 628+0+0 (secure 0 0 0) 0x7f34ec02cb70 con 0x7f34fc101700 2024-03-30T14:32:34.392 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:34.395+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=830}) -- 0x7f34e407a430 con 0x7f34fc101700 2024-03-30T14:32:34.395 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:34.398+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v829) ==== 217+0+0 (secure 0 0 0) 0x7f34ec089160 con 0x7f34fc101700 2024-03-30T14:32:34.502 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:34.505+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"} v 0) -- 0x7f34fc109c70 con 0x7f34fc101700 2024-03-30T14:32:34.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:34 smithi138 ceph-mon[33397]: pgmap v1813: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:34.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:34 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1198612153' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:34.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:34 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:34.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:34 smithi102 ceph-mon[29392]: pgmap v1813: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:34.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:34 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1198612153' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:34.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:34 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:34.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:34 smithi102 ceph-mon[37717]: pgmap v1813: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:34.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:34 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1198612153' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:34.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:34 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:35.397 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.400+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 9 ==== osd_map(830..830 src has 266..830) ==== 628+0+0 (secure 0 0 0) 0x7f34ec07e930 con 0x7f34fc101700 2024-03-30T14:32:35.397 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.400+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=831}) -- 0x7f34e407a8a0 con 0x7f34fc101700 2024-03-30T14:32:35.402 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.405+0000 7f34f97fa640 1 -- 172.21.15.102:0/1198612153 <== mon.2 v2:172.21.15.102:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v830) ==== 217+0+0 (secure 0 0 0) 0x7f34ec0b17e0 con 0x7f34fc101700 2024-03-30T14:32:35.402 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_bytes = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:32:35.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 >> v2:172.21.15.102:6800/622906077 conn(0x7f34e406db70 msgr2=0x7f34e4070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:35.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 --2- 172.21.15.102:0/1198612153 >> v2:172.21.15.102:6800/622906077 conn(0x7f34e406db70 0x7f34e4070030 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0ff930 tx=0x7f34f000b410 comp rx=0 tx=0).stop 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc101700 msgr2=0x7f34fc197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc101700 0x7f34fc197760 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f34ec000c00 tx=0x7f34ec034840 comp rx=0 tx=0).stop 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 shutdown_connections 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 --2- 172.21.15.102:0/1198612153 >> v2:172.21.15.102:6800/622906077 conn(0x7f34e406db70 0x7f34e4070030 unknown :-1 s=CLOSED pgs=3500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34fc107710 0x7f34fc19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f34fc1020c0 0x7f34fc197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 --2- 172.21.15.102:0/1198612153 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f34fc101700 0x7f34fc197760 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 >> 172.21.15.102:0/1198612153 conn(0x7f34fc0fd210 msgr2=0x7f34fc0fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 shutdown_connections 2024-03-30T14:32:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.409+0000 7f3502472640 1 -- 172.21.15.102:0/1198612153 wait complete. 2024-03-30T14:32:35.418 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool set-quota b802dd6e-da88-4515-b4f3-9f10759279d2 max_objects 0 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 -- 172.21.15.102:0/2416340166 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 msgr2=0x7f353810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2416340166 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 0x7f353810b7c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f3528009900 tx=0x7f352802bcc0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 -- 172.21.15.102:0/2416340166 shutdown_connections 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2416340166 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f353810bd90 0x7f353810e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2416340166 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 0x7f353810b7c0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2416340166 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 0x7f3538102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 -- 172.21.15.102:0/2416340166 >> 172.21.15.102:0/2416340166 conn(0x7f3538077830 msgr2=0x7f3538077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 -- 172.21.15.102:0/2416340166 shutdown_connections 2024-03-30T14:32:35.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.508+0000 7f353eaf4640 1 -- 172.21.15.102:0/2416340166 wait complete. 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 Processor -- start 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 -- start start 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 0x7f3538100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 0x7f3538100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f353810bd90 0x7f3538101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3538111230 con 0x7f35381032e0 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35381110b0 con 0x7f353810bd90 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353eaf4640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f35381113b0 con 0x7f35381029c0 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 0x7f3538100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 0x7f3538100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:33180/0 (socket says 172.21.15.102:33180) 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f352ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 0x7f3538100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 -- 172.21.15.102:0/2854328419 learned_addr learned my addr 172.21.15.102:0/2854328419 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353d091640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f353810bd90 0x7f3538101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 -- 172.21.15.102:0/2854328419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f353810bd90 msgr2=0x7f3538101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f353810bd90 0x7f3538101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 -- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 msgr2=0x7f3538100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 0x7f3538100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.509+0000 7f353c890640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35381a5c00 con 0x7f35381029c0 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.510+0000 7f352ffff640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 0x7f3538100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.510+0000 7f353c890640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 0x7f3538100610 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f353400ea60 tx=0x7f353400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.510+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353400cde0 con 0x7f35381029c0 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.510+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f35340035f0 con 0x7f35381029c0 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.510+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35381a5ef0 con 0x7f35381029c0 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.510+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534010690 con 0x7f35381029c0 2024-03-30T14:32:35.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.510+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381a6350 con 0x7f35381029c0 2024-03-30T14:32:35.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.511+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3538068a80 con 0x7f35381029c0 2024-03-30T14:32:35.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.511+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f3534004000 con 0x7f35381029c0 2024-03-30T14:32:35.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.512+0000 7f352dffb640 1 --2- 172.21.15.102:0/2854328419 >> v2:172.21.15.102:6800/622906077 conn(0x7f352006db70 0x7f3520070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:32:35.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.512+0000 7f352ffff640 1 --2- 172.21.15.102:0/2854328419 >> v2:172.21.15.102:6800/622906077 conn(0x7f352006db70 0x7f3520070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:32:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.512+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(830..830 src has 266..830) ==== 8194+0+0 (secure 0 0 0) 0x7f353408e6e0 con 0x7f35381029c0 2024-03-30T14:32:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.512+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=831}) -- 0x7f3520079880 con 0x7f35381029c0 2024-03-30T14:32:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.512+0000 7f352ffff640 1 --2- 172.21.15.102:0/2854328419 >> v2:172.21.15.102:6800/622906077 conn(0x7f352006db70 0x7f3520070030 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f3528006010 tx=0x7f3528005e40 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:32:35.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.516+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f35340042f0 con 0x7f35381029c0 2024-03-30T14:32:35.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:32:35.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:35 smithi138 ceph-mon[33397]: osdmap e829: 8 total, 8 up, 8 in 2024-03-30T14:32:35.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:35 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/1198612153' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:35.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:35 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[37717]: osdmap e829: 8 total, 8 up, 8 in 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/1198612153' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[29392]: osdmap e829: 8 total, 8 up, 8 in 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/1198612153' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:35.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:35 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-30T14:32:35.690 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:35.692+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"} v 0) -- 0x7f3538106590 con 0x7f35381029c0 2024-03-30T14:32:36.399 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:36.402+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 7 ==== osd_map(831..831 src has 266..831) ==== 628+0+0 (secure 0 0 0) 0x7f3534059380 con 0x7f35381029c0 2024-03-30T14:32:36.399 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:36.402+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=832}) -- 0x7f352007a3d0 con 0x7f35381029c0 2024-03-30T14:32:36.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:36.407+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v831) ==== 221+0+0 (secure 0 0 0) 0x7f353407b0b0 con 0x7f35381029c0 2024-03-30T14:32:36.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:36.515+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"} v 0) -- 0x7f3538106000 con 0x7f35381029c0 2024-03-30T14:32:36.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:36 smithi138 ceph-mon[33397]: pgmap v1815: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:36.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:32:36.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:36 smithi138 ceph-mon[33397]: osdmap e830: 8 total, 8 up, 8 in 2024-03-30T14:32:36.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:36 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2854328419' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:36.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:36 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:36.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:32:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:32:36.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[29392]: pgmap v1815: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:36.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:32:36.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[29392]: osdmap e830: 8 total, 8 up, 8 in 2024-03-30T14:32:36.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2854328419' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:36.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:36.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[37717]: pgmap v1815: 188 pgs: 188 active+clean; 859 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:36.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_bytes", "val": "0"}]': finished 2024-03-30T14:32:36.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[37717]: osdmap e830: 8 total, 8 up, 8 in 2024-03-30T14:32:36.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2854328419' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:36.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:36 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:37.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.416+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 9 ==== osd_map(832..832 src has 266..832) ==== 628+0+0 (secure 0 0 0) 0x7f353408c2e0 con 0x7f35381029c0 2024-03-30T14:32:37.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.416+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=833}) -- 0x7f352007aa10 con 0x7f35381029c0 2024-03-30T14:32:37.418 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.420+0000 7f352dffb640 1 -- 172.21.15.102:0/2854328419 <== mon.2 v2:172.21.15.102:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 v832) ==== 221+0+0 (secure 0 0 0) 0x7f353405b9e0 con 0x7f35381029c0 2024-03-30T14:32:37.418 INFO:tasks.workunit.client.0.smithi102.stderr:set-quota max_objects = 0 for pool b802dd6e-da88-4515-b4f3-9f10759279d2 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 >> v2:172.21.15.102:6800/622906077 conn(0x7f352006db70 msgr2=0x7f3520070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2854328419 >> v2:172.21.15.102:6800/622906077 conn(0x7f352006db70 0x7f3520070030 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f3528006010 tx=0x7f3528005e40 comp rx=0 tx=0).stop 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 msgr2=0x7f3538100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 0x7f3538100610 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f353400ea60 tx=0x7f353400ef20 comp rx=0 tx=0).stop 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 shutdown_connections 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2854328419 >> v2:172.21.15.102:6800/622906077 conn(0x7f352006db70 0x7f3520070030 unknown :-1 s=CLOSED pgs=3501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f353810bd90 0x7f3538101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35381032e0 0x7f3538100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 --2- 172.21.15.102:0/2854328419 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35381029c0 0x7f3538100610 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:32:37.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.424+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 >> 172.21.15.102:0/2854328419 conn(0x7f3538077830 msgr2=0x7f3538103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:32:37.423 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.425+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 shutdown_connections 2024-03-30T14:32:37.423 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:32:37.425+0000 7f353eaf4640 1 -- 172.21.15.102:0/2854328419 wait complete. 2024-03-30T14:32:37.433 INFO:tasks.workunit.client.0.smithi102.stderr:+ sleep 30 2024-03-30T14:32:37.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:32:37 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:32:37.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:32:37.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:37 smithi138 ceph-mon[33397]: osdmap e831: 8 total, 8 up, 8 in 2024-03-30T14:32:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:37 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2854328419' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:37.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:37 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:37.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:32:37.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[29392]: osdmap e831: 8 total, 8 up, 8 in 2024-03-30T14:32:37.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2854328419' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:37.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:37.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:32:37.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[37717]: osdmap e831: 8 total, 8 up, 8 in 2024-03-30T14:32:37.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2854328419' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:37.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:37 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]: dispatch 2024-03-30T14:32:38.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:38 smithi138 ceph-mon[33397]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-30T14:32:38.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:38 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:38.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:38 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:32:38.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:38 smithi138 ceph-mon[33397]: osdmap e832: 8 total, 8 up, 8 in 2024-03-30T14:32:38.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[29392]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-30T14:32:38.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:38.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:32:38.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[29392]: osdmap e832: 8 total, 8 up, 8 in 2024-03-30T14:32:38.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[37717]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-30T14:32:38.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:38.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "field": "max_objects", "val": "0"}]': finished 2024-03-30T14:32:38.690 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:38 smithi102 ceph-mon[37717]: osdmap e832: 8 total, 8 up, 8 in 2024-03-30T14:32:40.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:40 smithi138 ceph-mon[33397]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-30T14:32:40.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:40 smithi102 ceph-mon[29392]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-30T14:32:40.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:40 smithi102 ceph-mon[37717]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-30T14:32:42.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:42 smithi138 ceph-mon[33397]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-30T14:32:42.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:42 smithi102 ceph-mon[29392]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-30T14:32:42.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:42 smithi102 ceph-mon[37717]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-30T14:32:43.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:43 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:43.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:43 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:43.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:43 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:44.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:44 smithi138 ceph-mon[33397]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-30T14:32:44.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:44 smithi102 ceph-mon[29392]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-30T14:32:44.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:44 smithi102 ceph-mon[37717]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-30T14:32:46.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:46 smithi138 ceph-mon[33397]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 601 B/s rd, 0 op/s 2024-03-30T14:32:46.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:46 smithi102 ceph-mon[29392]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 601 B/s rd, 0 op/s 2024-03-30T14:32:46.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:46 smithi102 ceph-mon[37717]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 601 B/s rd, 0 op/s 2024-03-30T14:32:46.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:32:46 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:32:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:32:47.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:32:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:32:48.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:48 smithi138 ceph-mon[33397]: pgmap v1824: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:48.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:48 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:48.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:48 smithi102 ceph-mon[29392]: pgmap v1824: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:48.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:48 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:48.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:48 smithi102 ceph-mon[37717]: pgmap v1824: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:32:48.689 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:48 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:50.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:50 smithi138 ceph-mon[33397]: pgmap v1825: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-03-30T14:32:50.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:50 smithi102 ceph-mon[29392]: pgmap v1825: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-03-30T14:32:50.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:50 smithi102 ceph-mon[37717]: pgmap v1825: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-03-30T14:32:51.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:51 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:32:51.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:51.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:51 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:51.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:51 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:32:51.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:51.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:51 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:51.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:51 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:32:51.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:51.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:51 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:52.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:52 smithi138 ceph-mon[33397]: pgmap v1826: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:52.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:52 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:32:52.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:52 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:32:52.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:52 smithi138 ceph-mon[33397]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:52.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[29392]: pgmap v1826: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:52.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:32:52.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:32:52.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[29392]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:52.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[37717]: pgmap v1826: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:52.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:32:52.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:32:52.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:52 smithi102 ceph-mon[37717]: from='mgr.24464 ' entity='mgr.y' 2024-03-30T14:32:54.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:54 smithi138 ceph-mon[33397]: pgmap v1827: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:32:54.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:54 smithi102 ceph-mon[29392]: pgmap v1827: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:32:54.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:54 smithi102 ceph-mon[37717]: pgmap v1827: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:32:56.741 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:56 smithi102 ceph-mon[29392]: pgmap v1828: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:56.742 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:32:56 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:32:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:32:56.742 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:56 smithi102 ceph-mon[37717]: pgmap v1828: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:56.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:56 smithi138 ceph-mon[33397]: pgmap v1828: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:32:57.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:32:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:32:58.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:58 smithi138 ceph-mon[33397]: pgmap v1829: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:58.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:58 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:58.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:32:58 smithi138 ceph-mon[33397]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:58.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:58 smithi102 ceph-mon[29392]: pgmap v1829: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:58.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:58 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:58.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:32:58 smithi102 ceph-mon[29392]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:32:58.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:58 smithi102 ceph-mon[37717]: pgmap v1829: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:32:58.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:58 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:32:58.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:32:58 smithi102 ceph-mon[37717]: from='mgr.24464 172.21.15.102:0/2736528181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:33:00.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:00 smithi138 ceph-mon[33397]: pgmap v1830: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:33:00.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:00 smithi102 ceph-mon[29392]: pgmap v1830: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:33:00.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:00 smithi102 ceph-mon[37717]: pgmap v1830: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:33:02.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:02 smithi138 ceph-mon[33397]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:33:02.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:02 smithi102 ceph-mon[29392]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:33:02.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:02 smithi102 ceph-mon[37717]: pgmap v1831: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:33:04.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:04 smithi138 ceph-mon[33397]: pgmap v1832: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:33:04.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:04 smithi102 ceph-mon[29392]: pgmap v1832: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:33:04.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:04 smithi102 ceph-mon[37717]: pgmap v1832: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:33:06.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:06 smithi138 ceph-mon[33397]: pgmap v1833: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:33:06.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:06 smithi102 ceph-mon[29392]: pgmap v1833: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:33:06.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:06 smithi102 ceph-mon[37717]: pgmap v1833: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:33:06.939 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:06 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: ::ffff:172.21.15.138 - - [30/Mar/2024:14:33:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:33:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool delete b802dd6e-da88-4515-b4f3-9f10759279d2 b802dd6e-da88-4515-b4f3-9f10759279d2 --yes-i-really-really-mean-it 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.530+0000 7f113e316640 1 -- 172.21.15.102:0/2871679125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 msgr2=0x7f1138100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.530+0000 7f113e316640 1 --2- 172.21.15.102:0/2871679125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138100a90 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f112c009930 tx=0x7f112c02bc30 comp rx=0 tx=0).stop 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 -- 172.21.15.102:0/2871679125 shutdown_connections 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 --2- 172.21.15.102:0/2871679125 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1138100fd0 0x7f113810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 --2- 172.21.15.102:0/2871679125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138100a90 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 --2- 172.21.15.102:0/2871679125 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11381066b0 0x7f1138106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 -- 172.21.15.102:0/2871679125 >> 172.21.15.102:0/2871679125 conn(0x7f1138077770 msgr2=0x7f1138077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 -- 172.21.15.102:0/2871679125 shutdown_connections 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 -- 172.21.15.102:0/2871679125 wait complete. 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 Processor -- start 2024-03-30T14:33:07.528 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.531+0000 7f113e316640 1 -- start start 2024-03-30T14:33:07.531 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1138100fd0 0x7f1138196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11381066b0 0x7f113819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1138110100 con 0x7f1138100610 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f113810ff80 con 0x7f11381066b0 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1138110280 con 0x7f1138100fd0 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f1137fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1138100fd0 0x7f1138196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1138100fd0 0x7f1138196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:42604/0 (socket says 172.21.15.102:42604) 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 -- 172.21.15.102:0/2618985452 learned_addr learned my addr 172.21.15.102:0/2618985452 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:33:07.532 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f1137fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44464/0 (socket says 172.21.15.102:44464) 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113c8b3640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11381066b0 0x7f113819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 -- 172.21.15.102:0/2618985452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11381066b0 msgr2=0x7f113819adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11381066b0 0x7f113819adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 -- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 msgr2=0x7f1138196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113819b4f0 con 0x7f1138100fd0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f1137fff640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11377fe640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1138100fd0 0x7f1138196a40 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f112c009230 tx=0x7f112c02c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c043020 con 0x7f1138100fd0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f112c0043d0 con 0x7f1138100fd0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c003b70 con 0x7f1138100fd0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113819b780 con 0x7f1138100fd0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.532+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11381a3070 con 0x7f1138100fd0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.534+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1138068630 con 0x7f1138100fd0 2024-03-30T14:33:07.533 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.534+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f112c046020 con 0x7f1138100fd0 2024-03-30T14:33:07.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.534+0000 7f11357fa640 1 --2- 172.21.15.102:0/2618985452 >> v2:172.21.15.102:6800/622906077 conn(0x7f112006db70 0x7f1120070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:07.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.534+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(832..832 src has 266..832) ==== 8194+0+0 (secure 0 0 0) 0x7f112c0afdf0 con 0x7f1138100fd0 2024-03-30T14:33:07.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.534+0000 7f1137fff640 1 --2- 172.21.15.102:0/2618985452 >> v2:172.21.15.102:6800/622906077 conn(0x7f112006db70 0x7f1120070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:07.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.534+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=833}) -- 0x7f1120079880 con 0x7f1138100fd0 2024-03-30T14:33:07.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.535+0000 7f1137fff640 1 --2- 172.21.15.102:0/2618985452 >> v2:172.21.15.102:6800/622906077 conn(0x7f112006db70 0x7f1120070030 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f1138101750 tx=0x7f1128005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:33:07.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.539+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f112c04d070 con 0x7f1138100fd0 2024-03-30T14:33:07.685 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:33:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug there is no tcmu-runner data available 2024-03-30T14:33:07.709 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:07.713+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1138108b80 con 0x7f1138100fd0 2024-03-30T14:33:08.470 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.474+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 7 ==== osd_map(833..833 src has 266..833) ==== 296+0+0 (secure 0 0 0) 0x7f112c07ab40 con 0x7f1138100fd0 2024-03-30T14:33:08.471 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.474+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=834}) -- 0x7f112007a150 con 0x7f1138100fd0 2024-03-30T14:33:08.474 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.477+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]=0 pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' removed v833) ==== 248+0+0 (secure 0 0 0) 0x7f112c080b40 con 0x7f1138100fd0 2024-03-30T14:33:08.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.592+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true} v 0) -- 0x7f113802d300 con 0x7f1138100fd0 2024-03-30T14:33:08.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.593+0000 7f11357fa640 1 -- 172.21.15.102:0/2618985452 <== mon.2 v2:172.21.15.102:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]=0 pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' does not exist v833) ==== 255+0+0 (secure 0 0 0) 0x7f112c085370 con 0x7f1138100fd0 2024-03-30T14:33:08.590 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'b802dd6e-da88-4515-b4f3-9f10759279d2' does not exist 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 >> v2:172.21.15.102:6800/622906077 conn(0x7f112006db70 msgr2=0x7f1120070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 --2- 172.21.15.102:0/2618985452 >> v2:172.21.15.102:6800/622906077 conn(0x7f112006db70 0x7f1120070030 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f1138101750 tx=0x7f1128005f90 comp rx=0 tx=0).stop 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1138100fd0 msgr2=0x7f1138196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1138100fd0 0x7f1138196a40 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f112c009230 tx=0x7f112c02c790 comp rx=0 tx=0).stop 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 shutdown_connections 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 --2- 172.21.15.102:0/2618985452 >> v2:172.21.15.102:6800/622906077 conn(0x7f112006db70 0x7f1120070030 unknown :-1 s=CLOSED pgs=3502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11381066b0 0x7f113819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1138100fd0 0x7f1138196a40 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 --2- 172.21.15.102:0/2618985452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1138100610 0x7f1138196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 >> 172.21.15.102:0/2618985452 conn(0x7f1138077770 msgr2=0x7f11380ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:33:08.593 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 shutdown_connections 2024-03-30T14:33:08.593 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.595+0000 7f113e316640 1 -- 172.21.15.102:0/2618985452 wait complete. 2024-03-30T14:33:08.600 INFO:tasks.workunit.client.0.smithi102.stderr:+ ceph osd pool delete d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 d4c70a77-bc23-4cc8-a2e8-5c3b468478c1 --yes-i-really-really-mean-it 2024-03-30T14:33:08.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.686+0000 7f092697a640 1 -- 172.21.15.102:0/654264867 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f092010bd00 msgr2=0x7f092010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:08.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.686+0000 7f092697a640 1 --2- 172.21.15.102:0/654264867 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f092010bd00 0x7f092010e1a0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f0918009ca0 tx=0x7f091802c230 comp rx=0 tx=0).stop 2024-03-30T14:33:08.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.686+0000 7f092697a640 1 -- 172.21.15.102:0/654264867 shutdown_connections 2024-03-30T14:33:08.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.686+0000 7f092697a640 1 --2- 172.21.15.102:0/654264867 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f092010bd00 0x7f092010e1a0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.686+0000 7f092697a640 1 --2- 172.21.15.102:0/654264867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09201032e0 0x7f092010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.686+0000 7f092697a640 1 --2- 172.21.15.102:0/654264867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09201029c0 0x7f0920102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.686+0000 7f092697a640 1 -- 172.21.15.102:0/654264867 >> 172.21.15.102:0/654264867 conn(0x7f0920077770 msgr2=0x7f0920077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 -- 172.21.15.102:0/654264867 shutdown_connections 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 -- 172.21.15.102:0/654264867 wait complete. 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 Processor -- start 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 -- start start 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09201029c0 0x7f09201976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09201032e0 0x7f0920197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f092010bd00 0x7f092019bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0920111210 con 0x7f092010bd00 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0920111090 con 0x7f09201029c0 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.687+0000 7f092697a640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0920111390 con 0x7f09201032e0 2024-03-30T14:33:08.686 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09201032e0 0x7f0920197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f0924f17640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f092010bd00 0x7f092019bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09201032e0 0x7f0920197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:42622/0 (socket says 172.21.15.102:42622) 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 -- 172.21.15.102:0/3929847113 learned_addr learned my addr 172.21.15.102:0/3929847113 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091ffff640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09201029c0 0x7f09201976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 -- 172.21.15.102:0/3929847113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09201029c0 msgr2=0x7f09201976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09201029c0 0x7f09201976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 -- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f092010bd00 msgr2=0x7f092019bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f092010bd00 0x7f092019bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092019c6b0 con 0x7f09201032e0 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f0924f17640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f092010bd00 0x7f092019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091f7fe640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09201032e0 0x7f0920197c00 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f091400d770 tx=0x7f091400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914017070 con 0x7f09201032e0 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092019c9a0 con 0x7f09201032e0 2024-03-30T14:33:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(26 keys) ==== 1093+0+0 (secure 0 0 0) 0x7f09140054d0 con 0x7f09201032e0 2024-03-30T14:33:08.688 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914004cb0 con 0x7f09201032e0 2024-03-30T14:33:08.688 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.688+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09201a4230 con 0x7f09201032e0 2024-03-30T14:33:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.689+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0920068630 con 0x7f09201032e0 2024-03-30T14:33:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.690+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 21) ==== 91798+0+0 (secure 0 0 0) 0x7f0914004e50 con 0x7f09201032e0 2024-03-30T14:33:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.690+0000 7f091d7fa640 1 --2- 172.21.15.102:0/3929847113 >> v2:172.21.15.102:6800/622906077 conn(0x7f090806db70 0x7f0908070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T14:33:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.690+0000 7f091ffff640 1 --2- 172.21.15.102:0/3929847113 >> v2:172.21.15.102:6800/622906077 conn(0x7f090806db70 0x7f0908070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T14:33:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.690+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(833..833 src has 266..833) ==== 7806+0+0 (secure 0 0 0) 0x7f091408db10 con 0x7f09201032e0 2024-03-30T14:33:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.690+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=834}) -- 0x7f0908079060 con 0x7f09201032e0 2024-03-30T14:33:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.691+0000 7f091ffff640 1 --2- 172.21.15.102:0/3929847113 >> v2:172.21.15.102:6800/622906077 conn(0x7f090806db70 0x7f0908070030 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f09100095f0 tx=0x7f0910005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T14:33:08.691 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.695+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f091405ea40 con 0x7f09201032e0 2024-03-30T14:33:08.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:08.869+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0920106590 con 0x7f09201032e0 2024-03-30T14:33:08.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:08 smithi138 ceph-mon[33397]: pgmap v1834: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:33:08.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:08 smithi138 ceph-mon[33397]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:33:08.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:08 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2618985452' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:08.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:08 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:08.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[29392]: pgmap v1834: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:33:08.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[29392]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:33:08.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2618985452' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:08.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:08.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[37717]: pgmap v1834: 188 pgs: 188 active+clean; 871 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:33:08.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[37717]: from='client.24437 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:33:08.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2618985452' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:08.940 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:08 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.492 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.496+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 7 ==== osd_map(834..834 src has 266..834) ==== 296+0+0 (secure 0 0 0) 0x7f091408b920 con 0x7f09201032e0 2024-03-30T14:33:09.493 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.496+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=835}) -- 0x7f09080798f0 con 0x7f09201032e0 2024-03-30T14:33:09.498 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.502+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]=0 pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' removed v834) ==== 248+0+0 (secure 0 0 0) 0x7f0914063270 con 0x7f09201032e0 2024-03-30T14:33:09.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.615+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true} v 0) -- 0x7f09201061d0 con 0x7f09201032e0 2024-03-30T14:33:09.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.616+0000 7f091d7fa640 1 -- 172.21.15.102:0/3929847113 <== mon.2 v2:172.21.15.102:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]=0 pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' does not exist v834) ==== 255+0+0 (secure 0 0 0) 0x7f0914058a40 con 0x7f09201032e0 2024-03-30T14:33:09.612 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'd4c70a77-bc23-4cc8-a2e8-5c3b468478c1' does not exist 2024-03-30T14:33:09.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 >> v2:172.21.15.102:6800/622906077 conn(0x7f090806db70 msgr2=0x7f0908070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:09.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 --2- 172.21.15.102:0/3929847113 >> v2:172.21.15.102:6800/622906077 conn(0x7f090806db70 0x7f0908070030 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f09100095f0 tx=0x7f0910005f90 comp rx=0 tx=0).stop 2024-03-30T14:33:09.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09201032e0 msgr2=0x7f0920197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09201032e0 0x7f0920197c00 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f091400d770 tx=0x7f091400dc30 comp rx=0 tx=0).stop 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 shutdown_connections 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 --2- 172.21.15.102:0/3929847113 >> v2:172.21.15.102:6800/622906077 conn(0x7f090806db70 0x7f0908070030 unknown :-1 s=CLOSED pgs=3503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f092010bd00 0x7f092019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f09201032e0 0x7f0920197c00 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 --2- 172.21.15.102:0/3929847113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09201029c0 0x7f09201976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 >> 172.21.15.102:0/3929847113 conn(0x7f0920077770 msgr2=0x7f0920103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 shutdown_connections 2024-03-30T14:33:09.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-03-30T14:33:09.618+0000 7f092697a640 1 -- 172.21.15.102:0/3929847113 wait complete. 2024-03-30T14:33:09.639 INFO:tasks.workunit.client.0.smithi102.stderr:+ echo OK 2024-03-30T14:33:09.640 INFO:tasks.workunit.client.0.smithi102.stdout:OK 2024-03-30T14:33:09.640 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T14:33:09.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T14:33:09.710 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-30T14:33:09.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-30T14:33:09.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:33:09.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[29392]: osdmap e833: 8 total, 8 up, 8 in 2024-03-30T14:33:09.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/2618985452' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.774 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3929847113' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[37717]: osdmap e833: 8 total, 8 up, 8 in 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/2618985452' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3929847113' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.775 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:09 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:33:09.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:09 smithi138 ceph-mon[33397]: osdmap e833: 8 total, 8 up, 8 in 2024-03-30T14:33:09.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/2618985452' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "b802dd6e-da88-4515-b4f3-9f10759279d2", "pool2": "b802dd6e-da88-4515-b4f3-9f10759279d2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:09 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3929847113' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:09.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:09 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:10.035 DEBUG:teuthology.parallel:result is None 2024-03-30T14:33:10.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T14:33:10.102 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T14:33:10.103 DEBUG:teuthology.orchestra.run.smithi102:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-30T14:33:10.166 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T14:33:10.166 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-30T14:33:10.180 INFO:tasks.cephadm:Teardown begin 2024-03-30T14:33:10.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:33:10.234 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:33:10.269 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-30T14:33:10.270 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:33:10.289 DEBUG:teuthology.orchestra.run.smithi138:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:33:10.328 INFO:tasks.cephadm:Stopping all daemons... 2024-03-30T14:33:10.328 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-30T14:33:10.329 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T14:33:10.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[37717]: pgmap v1836: 176 pgs: 176 active+clean; 698 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:33:10.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[37717]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:33:10.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:33:10.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[37717]: osdmap e834: 8 total, 8 up, 8 in 2024-03-30T14:33:10.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[37717]: from='client.? 172.21.15.102:0/3929847113' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:10.642 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[37717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:10.642 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 systemd[1]: Stopping Ceph mon.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:10.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[29392]: pgmap v1836: 176 pgs: 176 active+clean; 698 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:33:10.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[29392]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:33:10.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:33:10.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[29392]: osdmap e834: 8 total, 8 up, 8 in 2024-03-30T14:33:10.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[29392]: from='client.? 172.21.15.102:0/3929847113' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:10.643 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-mon[29392]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:10.685 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:10 smithi138 ceph-mon[33397]: pgmap v1836: 176 pgs: 176 active+clean; 698 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:33:10.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:10 smithi138 ceph-mon[33397]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:33:10.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:33:10.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:10 smithi138 ceph-mon[33397]: osdmap e834: 8 total, 8 up, 8 in 2024-03-30T14:33:10.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:10 smithi138 ceph-mon[33397]: from='client.? 172.21.15.102:0/3929847113' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:10.686 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:10 smithi138 ceph-mon[33397]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "pool2": "d4c70a77-bc23-4cc8-a2e8-5c3b468478c1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:33:10.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a[29388]: 2024-03-30T14:33:10.645+0000 7fdb72b3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:10.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a[29388]: 2024-03-30T14:33:10.645+0000 7fdb72b3e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-30T14:33:10.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:10 smithi102 podman[152854]: 2024-03-30 14:33:10.675616935 +0000 UTC m=+0.161164502 container died 5062081f6df6a13272035cb1b3c6ad7aebc7b35cd284325315c6b521d11c60e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-30T14:33:11.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:11 smithi102 podman[152854]: 2024-03-30 14:33:11.095239381 +0000 UTC m=+0.580786949 container cleanup 5062081f6df6a13272035cb1b3c6ad7aebc7b35cd284325315c6b521d11c60e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-4e50bbc) 2024-03-30T14:33:11.189 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:11 smithi102 bash[152854]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T14:33:11.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service' 2024-03-30T14:33:11.587 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:11 smithi102 podman[152868]: 2024-03-30 14:33:11.330327827 +0000 UTC m=+0.655380025 container remove 5062081f6df6a13272035cb1b3c6ad7aebc7b35cd284325315c6b521d11c60e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-a, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-30T14:33:11.587 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:11 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-03-30T14:33:11.587 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:11 smithi102 systemd[1]: Stopped Ceph mon.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T14:33:11.587 INFO:journalctl@ceph.mon.a.smithi102.stdout:Mar 30 14:33:11 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.a.service: Consumed 2min 15.047s CPU time. 2024-03-30T14:33:11.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:11.603 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-30T14:33:11.603 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-30T14:33:11.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.c 2024-03-30T14:33:11.939 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:11 smithi102 systemd[1]: Stopping Ceph mon.c for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:12.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:11 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c[37713]: 2024-03-30T14:33:11.950+0000 7f66f4d8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:12.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:11 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c[37713]: 2024-03-30T14:33:11.950+0000 7f66f4d8c700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-30T14:33:12.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:12 smithi102 podman[152977]: 2024-03-30 14:33:12.042114818 +0000 UTC m=+0.212599838 container died f2c2a15c7568bb631b79415f6827aeeb541c6b956995174cf487c471eca0fd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-30T14:33:12.315 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:11 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:14:33:11] ENGINE Bus STOPPING 2024-03-30T14:33:12.315 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:11 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:14:33:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:33:12.315 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:11 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:14:33:11] ENGINE Bus STOPPED 2024-03-30T14:33:12.315 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:11 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:14:33:11] ENGINE Bus STARTING 2024-03-30T14:33:12.315 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:12 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:14:33:12] ENGINE Serving on http://:::9283 2024-03-30T14:33:12.315 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:12 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: [30/Mar/2024:14:33:12] ENGINE Bus STARTED 2024-03-30T14:33:12.648 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:12 smithi102 podman[152977]: 2024-03-30 14:33:12.425426166 +0000 UTC m=+0.595911182 container cleanup f2c2a15c7568bb631b79415f6827aeeb541c6b956995174cf487c471eca0fd78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:33:12.648 INFO:journalctl@ceph.mon.c.smithi102.stdout:Mar 30 14:33:12 smithi102 bash[152977]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-c 2024-03-30T14:33:12.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.c.service' 2024-03-30T14:33:12.823 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:12.823 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-30T14:33:12.823 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-30T14:33:12.823 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.b 2024-03-30T14:33:13.167 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:12 smithi138 systemd[1]: Stopping Ceph mon.b for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:13.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-b[33393]: 2024-03-30T14:33:13.169+0000 7f6989fdb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:13.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-b[33393]: 2024-03-30T14:33:13.169+0000 7f6989fdb700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-30T14:33:13.436 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:13 smithi138 podman[76140]: 2024-03-30 14:33:13.415487189 +0000 UTC m=+0.396400295 container died c162e19f0addcb465e87c7cc64eafd689428c5182ab5f567a4c1c0bd399645e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:33:14.136 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:13 smithi138 podman[76140]: 2024-03-30 14:33:13.888220231 +0000 UTC m=+0.869133328 container cleanup c162e19f0addcb465e87c7cc64eafd689428c5182ab5f567a4c1c0bd399645e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc) 2024-03-30T14:33:14.136 INFO:journalctl@ceph.mon.b.smithi138.stdout:Mar 30 14:33:13 smithi138 bash[76140]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mon-b 2024-03-30T14:33:14.302 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mon.b.service' 2024-03-30T14:33:14.348 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:14.348 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-30T14:33:14.348 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-30T14:33:14.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T14:33:14.689 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:14 smithi102 systemd[1]: Stopping Ceph mgr.y for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:15.053 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:14 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T14:33:14.729+0000 7ff4c84e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:15.053 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:14 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y[29622]: 2024-03-30T14:33:14.729+0000 7ff4c84e0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-30T14:33:15.053 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:14 smithi102 podman[153109]: 2024-03-30 14:33:14.761440625 +0000 UTC m=+0.220964778 container died 358299fce19e985026c5c10793060d21f81f8dd64afff573b67fee8a3f72f6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-30T14:33:15.345 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:15 smithi102 podman[153109]: 2024-03-30 14:33:15.147368295 +0000 UTC m=+0.606892448 container cleanup 358299fce19e985026c5c10793060d21f81f8dd64afff573b67fee8a3f72f6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-30T14:33:15.345 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 30 14:33:15 smithi102 bash[153109]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-y 2024-03-30T14:33:15.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-03-30T14:33:15.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:15.593 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-30T14:33:15.593 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-30T14:33:15.594 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.x 2024-03-30T14:33:15.919 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 14:33:15 smithi138 systemd[1]: Stopping Ceph mgr.x for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:16.185 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 14:33:15 smithi138 podman[76262]: 2024-03-30 14:33:15.922564607 +0000 UTC m=+0.135825867 container died c79e20d192f94021fe6cdf733f28d586b81d4eb7ee4bc6406cbb0beae3ed5240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-03-30T14:33:16.533 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 14:33:16 smithi138 podman[76262]: 2024-03-30 14:33:16.285481209 +0000 UTC m=+0.498742460 container cleanup c79e20d192f94021fe6cdf733f28d586b81d4eb7ee4bc6406cbb0beae3ed5240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-30T14:33:16.533 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Mar 30 14:33:16 smithi138 bash[76262]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-mgr-x 2024-03-30T14:33:16.656 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-03-30T14:33:16.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:16.693 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-30T14:33:16.693 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T14:33:16.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T14:33:17.006 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:16 smithi102 systemd[1]: Stopping Ceph osd.0 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:17.439 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0[41735]: 2024-03-30T14:33:17.009+0000 7faa479aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:17.439 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0[41735]: 2024-03-30T14:33:17.009+0000 7faa479aa700 -1 osd.0 834 *** Got signal Terminated *** 2024-03-30T14:33:17.440 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:17 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0[41735]: 2024-03-30T14:33:17.009+0000 7faa479aa700 -1 osd.0 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:22.689 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:22 smithi102 podman[153231]: 2024-03-30 14:33:22.408658502 +0000 UTC m=+5.522103131 container died 0af2b4d93b65ee48bc7a3de81d336afe84278c3e7157188515458c2428ef6c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-30T14:33:23.082 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:22 smithi102 podman[153231]: 2024-03-30 14:33:22.817962979 +0000 UTC m=+5.931407607 container cleanup 0af2b4d93b65ee48bc7a3de81d336afe84278c3e7157188515458c2428ef6c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:33:23.083 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:22 smithi102 bash[153231]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T14:33:23.417 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:23 smithi102 podman[153249]: 2024-03-30 14:33:23.085999579 +0000 UTC m=+0.676650666 container remove 0af2b4d93b65ee48bc7a3de81d336afe84278c3e7157188515458c2428ef6c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc) 2024-03-30T14:33:23.689 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:23 smithi102 podman[153340]: 2024-03-30 14:33:23.324640182 +0000 UTC m=+0.017789283 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:23.690 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:23 smithi102 podman[153340]: 2024-03-30 14:33:23.453672345 +0000 UTC m=+0.146821423 container create ad36e6639ee9b9d206eb1eeab0a626a1657e900083be5675d01031775b5ed1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-30T14:33:24.008 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:23 smithi102 podman[153340]: 2024-03-30 14:33:23.718124207 +0000 UTC m=+0.411273297 container init ad36e6639ee9b9d206eb1eeab0a626a1657e900083be5675d01031775b5ed1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-30T14:33:24.008 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:23 smithi102 podman[153340]: 2024-03-30 14:33:23.721639805 +0000 UTC m=+0.414788885 container start ad36e6639ee9b9d206eb1eeab0a626a1657e900083be5675d01031775b5ed1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0-deactivate, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:33:24.008 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:23 smithi102 podman[153340]: 2024-03-30 14:33:23.814806914 +0000 UTC m=+0.507956014 container attach ad36e6639ee9b9d206eb1eeab0a626a1657e900083be5675d01031775b5ed1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T14:33:24.274 INFO:journalctl@ceph.osd.0.smithi102.stdout:Mar 30 14:33:24 smithi102 podman[153340]: 2024-03-30 14:33:24.011814435 +0000 UTC m=+0.704963529 container died ad36e6639ee9b9d206eb1eeab0a626a1657e900083be5675d01031775b5ed1c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-03-30T14:33:24.624 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T14:33:24.666 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:24.666 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T14:33:24.666 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-30T14:33:24.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.1 2024-03-30T14:33:24.939 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:24 smithi102 systemd[1]: Stopping Ceph osd.1 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:25.247 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1[47444]: 2024-03-30T14:33:25.062+0000 7f0e59f98700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:25.247 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1[47444]: 2024-03-30T14:33:25.062+0000 7f0e59f98700 -1 osd.1 834 *** Got signal Terminated *** 2024-03-30T14:33:25.247 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:25 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1[47444]: 2024-03-30T14:33:25.062+0000 7f0e59f98700 -1 osd.1 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:30.690 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:30 smithi102 podman[153460]: 2024-03-30 14:33:30.426117259 +0000 UTC m=+5.515731891 container died 832c402c0f1ab3cb9f57198392a62a118a9c28557166419db48662f9e3feac4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-30T14:33:31.061 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:30 smithi102 podman[153460]: 2024-03-30 14:33:30.830568886 +0000 UTC m=+5.920183506 container cleanup 832c402c0f1ab3cb9f57198392a62a118a9c28557166419db48662f9e3feac4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T14:33:31.061 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:30 smithi102 bash[153460]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T14:33:31.382 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:31 smithi102 podman[153484]: 2024-03-30 14:33:31.065132637 +0000 UTC m=+0.642554754 container remove 832c402c0f1ab3cb9f57198392a62a118a9c28557166419db48662f9e3feac4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-30T14:33:31.634 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:31 smithi102 podman[153574]: 2024-03-30 14:33:31.289402666 +0000 UTC m=+0.018092064 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:31.635 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:31 smithi102 podman[153574]: 2024-03-30 14:33:31.399404094 +0000 UTC m=+0.128093478 container create 6707dbbb4c52356112ea998d83bf4ce7f4f3da5cd23f96ae7113f89f8b096409 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-30T14:33:31.635 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:31 smithi102 podman[153574]: 2024-03-30 14:33:31.635116365 +0000 UTC m=+0.363805764 container init 6707dbbb4c52356112ea998d83bf4ce7f4f3da5cd23f96ae7113f89f8b096409 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-30T14:33:31.635 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:31 smithi102 podman[153574]: 2024-03-30 14:33:31.638149193 +0000 UTC m=+0.366838590 container start 6707dbbb4c52356112ea998d83bf4ce7f4f3da5cd23f96ae7113f89f8b096409 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True) 2024-03-30T14:33:31.926 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:31 smithi102 podman[153574]: 2024-03-30 14:33:31.743596064 +0000 UTC m=+0.472285460 container attach 6707dbbb4c52356112ea998d83bf4ce7f4f3da5cd23f96ae7113f89f8b096409 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:33:32.189 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 30 14:33:31 smithi102 podman[153574]: 2024-03-30 14:33:31.929948014 +0000 UTC m=+0.658637403 container died 6707dbbb4c52356112ea998d83bf4ce7f4f3da5cd23f96ae7113f89f8b096409 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-30T14:33:32.518 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-30T14:33:32.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:32.563 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-30T14:33:32.563 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T14:33:32.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T14:33:32.889 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:32 smithi102 systemd[1]: Stopping Ceph osd.2 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:33.189 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:32 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2[53048]: 2024-03-30T14:33:32.890+0000 7fb3dbfb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:33.190 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:32 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2[53048]: 2024-03-30T14:33:32.890+0000 7fb3dbfb5700 -1 osd.2 834 *** Got signal Terminated *** 2024-03-30T14:33:33.190 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:32 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2[53048]: 2024-03-30T14:33:32.890+0000 7fb3dbfb5700 -1 osd.2 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:38.631 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:38 smithi102 podman[153679]: 2024-03-30 14:33:38.355605271 +0000 UTC m=+5.564120762 container died 1a731e6972d7f15ddc80e477c552ded98582872bcdeff9ad7fe1a3afaf8bcbe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-30T14:33:38.939 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:38 smithi102 podman[153679]: 2024-03-30 14:33:38.708037181 +0000 UTC m=+5.916552681 container cleanup 1a731e6972d7f15ddc80e477c552ded98582872bcdeff9ad7fe1a3afaf8bcbe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, ceph=True) 2024-03-30T14:33:38.940 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:38 smithi102 bash[153679]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T14:33:39.319 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:38 smithi102 podman[153696]: 2024-03-30 14:33:38.967548652 +0000 UTC m=+0.608966052 container remove 1a731e6972d7f15ddc80e477c552ded98582872bcdeff9ad7fe1a3afaf8bcbe4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T14:33:39.576 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:39 smithi102 podman[153785]: 2024-03-30 14:33:39.226036209 +0000 UTC m=+0.017468039 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:39.576 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:39 smithi102 podman[153785]: 2024-03-30 14:33:39.343464645 +0000 UTC m=+0.134896458 container create 51458f3eb106aaf377cbfe6ab569e4cad12fa91ebfd84b6908218ff4179a5c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-30T14:33:39.576 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:39 smithi102 podman[153785]: 2024-03-30 14:33:39.579450086 +0000 UTC m=+0.370881901 container init 51458f3eb106aaf377cbfe6ab569e4cad12fa91ebfd84b6908218ff4179a5c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:33:39.877 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:39 smithi102 podman[153785]: 2024-03-30 14:33:39.582736951 +0000 UTC m=+0.374168763 container start 51458f3eb106aaf377cbfe6ab569e4cad12fa91ebfd84b6908218ff4179a5c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-30T14:33:39.877 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:39 smithi102 podman[153785]: 2024-03-30 14:33:39.662652774 +0000 UTC m=+0.454084605 container attach 51458f3eb106aaf377cbfe6ab569e4cad12fa91ebfd84b6908218ff4179a5c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-03-30T14:33:40.170 INFO:journalctl@ceph.osd.2.smithi102.stdout:Mar 30 14:33:39 smithi102 podman[153785]: 2024-03-30 14:33:39.880699122 +0000 UTC m=+0.672130937 container died 51458f3eb106aaf377cbfe6ab569e4cad12fa91ebfd84b6908218ff4179a5c18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-30T14:33:40.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T14:33:40.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:40.480 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T14:33:40.480 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-30T14:33:40.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.3 2024-03-30T14:33:40.689 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:40 smithi102 systemd[1]: Stopping Ceph osd.3 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:41.189 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:40 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T14:33:40.792+0000 7f6b726c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:41.189 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:40 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T14:33:40.792+0000 7f6b726c1700 -1 osd.3 834 *** Got signal Terminated *** 2024-03-30T14:33:41.190 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:40 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T14:33:40.792+0000 7f6b726c1700 -1 osd.3 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:43.689 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:43 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T14:33:43.439+0000 7f6b6acad700 -1 osd.3 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.273272+0000 front 2024-03-30T14:33:18.273369+0000 (oldest deadline 2024-03-30T14:33:42.973369+0000) 2024-03-30T14:33:44.436 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:44.131+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:44.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:44 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:44.639+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:44.939 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:44 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T14:33:44.454+0000 7f6b6acad700 -1 osd.3 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.273272+0000 front 2024-03-30T14:33:18.273369+0000 (oldest deadline 2024-03-30T14:33:42.973369+0000) 2024-03-30T14:33:45.435 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:45.090+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:45.906 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:45.590+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:45.939 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:45 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3[59311]: 2024-03-30T14:33:45.452+0000 7f6b6acad700 -1 osd.3 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.273272+0000 front 2024-03-30T14:33:18.273369+0000 (oldest deadline 2024-03-30T14:33:42.973369+0000) 2024-03-30T14:33:46.185 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:46.097+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:33:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:33:45.909Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-03-30T14:33:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:33:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:33:45.912Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-03-30T14:33:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:33:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:33:45.912Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-03-30T14:33:46.186 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:33:45 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:33:45.912Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-03-30T14:33:46.540 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:46 smithi102 podman[153889]: 2024-03-30 14:33:46.214307502 +0000 UTC m=+5.515508788 container died 02a855c69bba64e171ce333bfd5ce447c88abeb9111159ffcdcd3534c650b5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-30T14:33:46.540 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:46 smithi102 podman[153889]: 2024-03-30 14:33:46.543218689 +0000 UTC m=+5.844419975 container cleanup 02a855c69bba64e171ce333bfd5ce447c88abeb9111159ffcdcd3534c650b5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-30T14:33:46.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:46.591+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:46.939 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:46 smithi102 bash[153889]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T14:33:46.939 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:46 smithi102 podman[153906]: 2024-03-30 14:33:46.744840782 +0000 UTC m=+0.528228720 container remove 02a855c69bba64e171ce333bfd5ce447c88abeb9111159ffcdcd3534c650b5ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:33:47.435 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:47.119+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:47.436 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:46 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:46.956+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:47.439 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:47 smithi102 podman[153998]: 2024-03-30 14:33:47.153856384 +0000 UTC m=+0.108800268 container create 2a5d83e1a63a740115353c1fa38e2bf11ec9978b4517b85198644d5be7b2542b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T14:33:47.439 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:47 smithi102 podman[153998]: 2024-03-30 14:33:47.062335431 +0000 UTC m=+0.017279317 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:47.440 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:47 smithi102 podman[153998]: 2024-03-30 14:33:47.381526626 +0000 UTC m=+0.336470526 container init 2a5d83e1a63a740115353c1fa38e2bf11ec9978b4517b85198644d5be7b2542b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-30T14:33:47.440 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:47 smithi102 podman[153998]: 2024-03-30 14:33:47.384852189 +0000 UTC m=+0.339796081 container start 2a5d83e1a63a740115353c1fa38e2bf11ec9978b4517b85198644d5be7b2542b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-30T14:33:47.906 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:47 smithi102 podman[153998]: 2024-03-30 14:33:47.467061082 +0000 UTC m=+0.422004986 container attach 2a5d83e1a63a740115353c1fa38e2bf11ec9978b4517b85198644d5be7b2542b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-30T14:33:47.906 INFO:journalctl@ceph.osd.3.smithi102.stdout:Mar 30 14:33:47 smithi102 podman[153998]: 2024-03-30 14:33:47.672212478 +0000 UTC m=+0.627156358 container died 2a5d83e1a63a740115353c1fa38e2bf11ec9978b4517b85198644d5be7b2542b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326) 2024-03-30T14:33:47.920 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:47.613+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:47.920 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:47.546+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:17.099710+0000 front 2024-03-30T14:33:17.099687+0000 (oldest deadline 2024-03-30T14:33:47.100191+0000) 2024-03-30T14:33:48.186 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:48 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:48.151+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:48.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:47 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:47.922+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:48.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-30T14:33:48.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:48.381 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-30T14:33:48.381 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T14:33:48.382 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T14:33:48.686 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:48 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:48.616+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:48.686 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:48 smithi138 systemd[1]: Stopping Ceph osd.4 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:48.686 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:48 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:48.541+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:17.099710+0000 front 2024-03-30T14:33:17.099687+0000 (oldest deadline 2024-03-30T14:33:47.100191+0000) 2024-03-30T14:33:49.183 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:48 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:48.893+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:49.183 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:48 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:48.714+0000 7fdb49ac9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:49.183 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:48 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:48.714+0000 7fdb49ac9700 -1 osd.4 834 *** Got signal Terminated *** 2024-03-30T14:33:49.184 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:48 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:48.714+0000 7fdb49ac9700 -1 osd.4 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:49.435 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:49 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:49.186+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:49.878 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:49 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:49.599+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:49.878 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:49 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:49.571+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:17.099710+0000 front 2024-03-30T14:33:17.099687+0000 (oldest deadline 2024-03-30T14:33:47.100191+0000) 2024-03-30T14:33:50.185 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:49 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:49.881+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:50.561 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:50 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:50.200+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:50.906 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:50 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:50.568+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:50.907 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:50 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:50.564+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:17.099710+0000 front 2024-03-30T14:33:17.099687+0000 (oldest deadline 2024-03-30T14:33:47.100191+0000) 2024-03-30T14:33:51.172 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:50 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:50.909+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:51.436 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:51.175+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:51.872 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:51.588+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:51.872 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:51.552+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:17.099710+0000 front 2024-03-30T14:33:17.099687+0000 (oldest deadline 2024-03-30T14:33:47.100191+0000) 2024-03-30T14:33:51.872 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:51.552+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.100696+0000 front 2024-03-30T14:33:27.100684+0000 (oldest deadline 2024-03-30T14:33:50.600610+0000) 2024-03-30T14:33:52.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:51.874+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:52.564 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:52 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:52.207+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:52.564 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:52 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:52.207+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:52.842 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:52 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:52.567+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:52.843 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:52 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:52.593+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:17.099710+0000 front 2024-03-30T14:33:17.099687+0000 (oldest deadline 2024-03-30T14:33:47.100191+0000) 2024-03-30T14:33:52.843 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:52 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:52.593+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.100696+0000 front 2024-03-30T14:33:27.100684+0000 (oldest deadline 2024-03-30T14:33:50.600610+0000) 2024-03-30T14:33:53.185 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:52 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:52.845+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:53.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:52 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:52.845+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:33:53.578 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:53 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:53.227+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:53.578 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:53 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:53.227+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:53.831 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:53 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:53.581+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:53.832 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:53 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:53.618+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:17.099710+0000 front 2024-03-30T14:33:17.099687+0000 (oldest deadline 2024-03-30T14:33:47.100191+0000) 2024-03-30T14:33:53.832 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:53 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4[38407]: 2024-03-30T14:33:53.618+0000 7fdb460bd700 -1 osd.4 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.100696+0000 front 2024-03-30T14:33:27.100684+0000 (oldest deadline 2024-03-30T14:33:50.600610+0000) 2024-03-30T14:33:54.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:53 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:53.834+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:54.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:53 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:53.834+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:33:54.521 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:54 smithi138 podman[76399]: 2024-03-30 14:33:54.269055052 +0000 UTC m=+5.687836531 container died d6ce33e0df35e79598d3efa3f8786110d0013a966626defdb1e60c6fc84c55c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc) 2024-03-30T14:33:54.521 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:54 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:54.221+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:54.521 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:54 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:54.221+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:54.846 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:54 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:54.614+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:54.846 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:54 smithi138 podman[76399]: 2024-03-30 14:33:54.614385662 +0000 UTC m=+6.033167149 container cleanup d6ce33e0df35e79598d3efa3f8786110d0013a966626defdb1e60c6fc84c55c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc) 2024-03-30T14:33:54.846 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:54 smithi138 bash[76399]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T14:33:55.186 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:54 smithi138 podman[76416]: 2024-03-30 14:33:54.907596088 +0000 UTC m=+0.641492412 container remove d6ce33e0df35e79598d3efa3f8786110d0013a966626defdb1e60c6fc84c55c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-30T14:33:55.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:54 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:54.848+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:55.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:54 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:54.848+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:33:55.567 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:55.250+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:55.567 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:55.250+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:55.568 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:55 smithi138 podman[76509]: 2024-03-30 14:33:55.144581623 +0000 UTC m=+0.018297580 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:33:55.568 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:55 smithi138 podman[76509]: 2024-03-30 14:33:55.276154073 +0000 UTC m=+0.149870031 container create 7c551b752906c7d3a2b1ae3c36c2e85b7d1928ff63409aba96808cd32004eb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-30T14:33:55.568 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:55 smithi138 podman[76509]: 2024-03-30 14:33:55.487379965 +0000 UTC m=+0.361095922 container init 7c551b752906c7d3a2b1ae3c36c2e85b7d1928ff63409aba96808cd32004eb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-30T14:33:55.568 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:55 smithi138 podman[76509]: 2024-03-30 14:33:55.490392709 +0000 UTC m=+0.364108676 container start 7c551b752906c7d3a2b1ae3c36c2e85b7d1928ff63409aba96808cd32004eb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-03-30T14:33:55.568 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:55 smithi138 podman[76509]: 2024-03-30 14:33:55.570810658 +0000 UTC m=+0.444526616 container attach 7c551b752906c7d3a2b1ae3c36c2e85b7d1928ff63409aba96808cd32004eb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:33:55.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:55.575+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:55.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:55.804+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:55.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:55 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:55.804+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:33:55.936 INFO:journalctl@ceph.osd.4.smithi138.stdout:Mar 30 14:33:55 smithi138 podman[76509]: 2024-03-30 14:33:55.785888782 +0000 UTC m=+0.659604739 container died 7c551b752906c7d3a2b1ae3c36c2e85b7d1928ff63409aba96808cd32004eb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-03-30T14:33:56.331 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:56.237+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:56.331 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:56.237+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:56.349 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T14:33:56.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:33:56.395 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T14:33:56.396 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-30T14:33:56.396 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.5 2024-03-30T14:33:56.613 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:56 smithi138 systemd[1]: Stopping Ceph osd.5 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:33:56.936 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:56.737+0000 7fc80578a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:33:56.936 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:56.737+0000 7fc80578a700 -1 osd.5 834 *** Got signal Terminated *** 2024-03-30T14:33:56.936 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:56.737+0000 7fc80578a700 -1 osd.5 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:33:56.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:56.757+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:56.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:56.757+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:33:56.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:56.615+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:56.937 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:56 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:56.615+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:33:57.587 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:57.188+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:57.587 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:57.188+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:57.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:57.769+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:57.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:57.769+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:33:57.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:57.598+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:57.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:57 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:57.598+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:33:58.607 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:58.237+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:58.607 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:58.237+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:58.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:58.719+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:58.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:58.719+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:33:58.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:58.609+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:58.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:58 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:58.609+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:33:59.644 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:59 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:59.212+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:33:59.644 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:33:59 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:33:59.212+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:33:59.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:59 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:59.646+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:33:59.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:33:59 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:33:59.647+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:33:59.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:59 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:59.716+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:33:59.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:33:59 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:33:59.716+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:00.436 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:00 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:34:00.168+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:34:00.436 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:00 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:34:00.168+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:34:00.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:00 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:00.648+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:00.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:00 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:00.648+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:00.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:00 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:00.751+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:00.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:00 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:00.751+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:00.937 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:00 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:00.751+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:01.436 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:01 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:34:01.139+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:19.224209+0000 front 2024-03-30T14:33:19.224288+0000 (oldest deadline 2024-03-30T14:33:43.324204+0000) 2024-03-30T14:34:01.436 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:01 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5[44096]: 2024-03-30T14:34:01.139+0000 7fc801d7e700 -1 osd.5 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:27.925791+0000 front 2024-03-30T14:33:27.925883+0000 (oldest deadline 2024-03-30T14:33:51.425751+0000) 2024-03-30T14:34:01.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:01 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:01.685+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:01.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:01 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:01.685+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:01.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:01 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:01.708+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:01.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:01 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:01.708+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:01.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:01 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:01.708+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:02.344 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:02 smithi138 podman[76615]: 2024-03-30 14:34:02.093098016 +0000 UTC m=+5.477282218 container died 85cf78ee6cd3fff76242b3b36b4b99d8c45d99e357e505b70176ce8203407a75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-30T14:34:02.674 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:02 smithi138 podman[76615]: 2024-03-30 14:34:02.55448151 +0000 UTC m=+5.938665720 container cleanup 85cf78ee6cd3fff76242b3b36b4b99d8c45d99e357e505b70176ce8203407a75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:34:02.674 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:02 smithi138 bash[76615]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T14:34:02.936 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:02 smithi138 podman[76643]: 2024-03-30 14:34:02.797911548 +0000 UTC m=+0.707635502 container remove 85cf78ee6cd3fff76242b3b36b4b99d8c45d99e357e505b70176ce8203407a75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-03-30T14:34:02.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:02.676+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:02.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:02.676+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:02.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:02.750+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:02.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:02.750+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:02.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:02.750+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:02.937 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:02 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:02.750+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:03.415 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:03 smithi138 podman[76738]: 2024-03-30 14:34:03.116078667 +0000 UTC m=+0.104071516 container create 74460ad0ed31097a503d005b62d19443677cf3a96e3999ee9f403083a593487b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-30T14:34:03.416 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:03 smithi138 podman[76738]: 2024-03-30 14:34:03.029469352 +0000 UTC m=+0.017462170 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:03.416 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:03 smithi138 podman[76738]: 2024-03-30 14:34:03.335488609 +0000 UTC m=+0.323481423 container init 74460ad0ed31097a503d005b62d19443677cf3a96e3999ee9f403083a593487b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T14:34:03.416 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:03 smithi138 podman[76738]: 2024-03-30 14:34:03.33874224 +0000 UTC m=+0.326735055 container start 74460ad0ed31097a503d005b62d19443677cf3a96e3999ee9f403083a593487b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:03.416 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:03 smithi138 podman[76738]: 2024-03-30 14:34:03.418709228 +0000 UTC m=+0.406702061 container attach 74460ad0ed31097a503d005b62d19443677cf3a96e3999ee9f403083a593487b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5-deactivate, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-30T14:34:03.683 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:03 smithi138 podman[76738]: 2024-03-30 14:34:03.631547547 +0000 UTC m=+0.619540365 container died 74460ad0ed31097a503d005b62d19443677cf3a96e3999ee9f403083a593487b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T14:34:03.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:03 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:03.685+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:03.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:03 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:03.685+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:03.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:03 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:03.685+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:37.870552+0000 front 2024-03-30T14:33:37.870452+0000 (oldest deadline 2024-03-30T14:34:03.170427+0000) 2024-03-30T14:34:03.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:03 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:03.772+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:03.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:03 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:03.772+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:03.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:03 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:03.772+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:03.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:03 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:03.772+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:04.235 INFO:journalctl@ceph.osd.5.smithi138.stdout:Mar 30 14:34:04 smithi138 podman[76765]: 2024-03-30 14:34:04.222899793 +0000 UTC m=+0.581805186 container remove 74460ad0ed31097a503d005b62d19443677cf3a96e3999ee9f403083a593487b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-30T14:34:04.240 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-30T14:34:04.280 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:34:04.281 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-30T14:34:04.281 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-30T14:34:04.281 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.6 2024-03-30T14:34:04.598 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:04 smithi138 systemd[1]: Stopping Ceph osd.6 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:04.598 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:04.600+0000 7f3e13c2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:34:04.598 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:04.600+0000 7f3e13c2e700 -1 osd.6 834 *** Got signal Terminated *** 2024-03-30T14:34:04.599 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:04.600+0000 7f3e13c2e700 -1 osd.6 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:34:04.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:04.724+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:04.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:04.724+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:04.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:04.724+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:37.870552+0000 front 2024-03-30T14:33:37.870452+0000 (oldest deadline 2024-03-30T14:34:03.170427+0000) 2024-03-30T14:34:04.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:04.738+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:04.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:04.738+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:04.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:04.738+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:04.937 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:04 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:04.738+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:06.186 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:05.694+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:06.186 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:05.694+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:06.186 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:05.694+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:37.870552+0000 front 2024-03-30T14:33:37.870452+0000 (oldest deadline 2024-03-30T14:34:03.170427+0000) 2024-03-30T14:34:06.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:05.776+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:06.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:05.776+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:06.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:05.776+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:06.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:05 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:05.777+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:07.082 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:06.718+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:07.082 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:06.718+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:07.082 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:06.718+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:37.870552+0000 front 2024-03-30T14:33:37.870452+0000 (oldest deadline 2024-03-30T14:34:03.170427+0000) 2024-03-30T14:34:07.083 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:06.774+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:07.083 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:06.774+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:07.083 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:06.774+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:07.083 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:06 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:06.774+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:07.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:07.738+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:07.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:07.738+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:07.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:07.738+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:07.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:07.738+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:07.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:07.737+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:07.937 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:07.737+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:07.937 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:07 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:07.737+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:37.870552+0000 front 2024-03-30T14:33:37.870452+0000 (oldest deadline 2024-03-30T14:34:03.170427+0000) 2024-03-30T14:34:09.186 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:08.736+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:18.568222+0000 front 2024-03-30T14:33:18.568365+0000 (oldest deadline 2024-03-30T14:33:44.468255+0000) 2024-03-30T14:34:09.186 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:08.736+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:24.468807+0000 front 2024-03-30T14:33:24.468685+0000 (oldest deadline 2024-03-30T14:33:56.269204+0000) 2024-03-30T14:34:09.186 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:08.736+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:37.870552+0000 front 2024-03-30T14:33:37.870452+0000 (oldest deadline 2024-03-30T14:34:03.170427+0000) 2024-03-30T14:34:09.186 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6[49797]: 2024-03-30T14:34:08.736+0000 7f3e10222700 -1 osd.6 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.170922+0000 front 2024-03-30T14:33:43.170906+0000 (oldest deadline 2024-03-30T14:34:08.371347+0000) 2024-03-30T14:34:09.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:08.705+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:09.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:08.705+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:09.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:08.705+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:09.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:08.705+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:09.187 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:08 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:08.705+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:09.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:09.662+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:09.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:09.662+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:09.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:09.662+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:09.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:09.662+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:09.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:09 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:09.662+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:10.300 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:10 smithi138 podman[76846]: 2024-03-30 14:34:10.046523216 +0000 UTC m=+5.541171348 container died 171d85cf1f1c98eff14ae7c877824bfe55b192c80f362d25365c59d96b590b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-30T14:34:10.666 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:10 smithi138 podman[76846]: 2024-03-30 14:34:10.40171827 +0000 UTC m=+5.896366401 container cleanup 171d85cf1f1c98eff14ae7c877824bfe55b192c80f362d25365c59d96b590b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_CLEAN=True) 2024-03-30T14:34:10.666 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:10 smithi138 bash[76846]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6 2024-03-30T14:34:10.936 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:10 smithi138 podman[76862]: 2024-03-30 14:34:10.669711692 +0000 UTC m=+0.618763145 container remove 171d85cf1f1c98eff14ae7c877824bfe55b192c80f362d25365c59d96b590b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:10.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:10 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:10.684+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:10.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:10 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:10.684+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:10.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:10 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:10.684+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:10.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:10 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:10.684+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:10.937 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:10 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:10.684+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:11.287 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:11 smithi138 podman[76955]: 2024-03-30 14:34:10.926883857 +0000 UTC m=+0.018218376 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:11.287 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:11 smithi138 podman[76955]: 2024-03-30 14:34:11.054169821 +0000 UTC m=+0.145504317 container create 8bb45fb1aa0b8980b3ad9c76b17b593790138ce3051f12ae9c59d52f557b0f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-30T14:34:11.287 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:11 smithi138 podman[76955]: 2024-03-30 14:34:11.290230715 +0000 UTC m=+0.381565231 container init 8bb45fb1aa0b8980b3ad9c76b17b593790138ce3051f12ae9c59d52f557b0f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc) 2024-03-30T14:34:11.580 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:11 smithi138 podman[76955]: 2024-03-30 14:34:11.293528824 +0000 UTC m=+0.384863327 container start 8bb45fb1aa0b8980b3ad9c76b17b593790138ce3051f12ae9c59d52f557b0f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:34:11.580 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:11 smithi138 podman[76955]: 2024-03-30 14:34:11.373951137 +0000 UTC m=+0.465285640 container attach 8bb45fb1aa0b8980b3ad9c76b17b593790138ce3051f12ae9c59d52f557b0f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-30T14:34:11.874 INFO:journalctl@ceph.osd.6.smithi138.stdout:Mar 30 14:34:11 smithi138 podman[76955]: 2024-03-30 14:34:11.58339415 +0000 UTC m=+0.674728657 container died 8bb45fb1aa0b8980b3ad9c76b17b593790138ce3051f12ae9c59d52f557b0f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-30T14:34:11.874 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:11.692+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:11.874 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:11.692+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:11.875 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:11.692+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:11.875 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:11.692+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:11.875 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:11 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:11.692+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:12.179 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.6.service' 2024-03-30T14:34:12.220 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:34:12.220 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-30T14:34:12.220 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-30T14:34:12.220 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.7 2024-03-30T14:34:12.435 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 systemd[1]: Stopping Ceph osd.7 for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.547+0000 7f789ca05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.547+0000 7f789ca05700 -1 osd.7 834 *** Got signal Terminated *** 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.547+0000 7f789ca05700 -1 osd.7 834 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.674+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.674+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.674+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.674+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:12.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:12 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:12.674+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:13.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:13.639+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:13.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:13.639+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:13.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:13.639+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:13.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:13.639+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:13.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:13 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:13.639+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:14.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:14 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:14.681+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:14.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:14 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:14.681+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:14.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:14 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:14.681+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:14.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:14 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:14.681+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:14.936 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:14 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:14.681+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:16.040 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:15.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:16.040 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:15.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:16.040 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:15.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:16.040 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:15.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:16.040 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:15 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:15.728+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:17.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:16.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6803 osd.0 since back 2024-03-30T14:33:21.413681+0000 front 2024-03-30T14:33:21.413704+0000 (oldest deadline 2024-03-30T14:33:46.713596+0000) 2024-03-30T14:34:17.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:16.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6807 osd.1 since back 2024-03-30T14:33:26.714282+0000 front 2024-03-30T14:33:26.714229+0000 (oldest deadline 2024-03-30T14:33:52.614169+0000) 2024-03-30T14:34:17.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:16.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6811 osd.2 since back 2024-03-30T14:33:35.415698+0000 front 2024-03-30T14:33:35.415650+0000 (oldest deadline 2024-03-30T14:34:00.115538+0000) 2024-03-30T14:34:17.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:16.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.102:6815 osd.3 since back 2024-03-30T14:33:43.517080+0000 front 2024-03-30T14:33:43.517131+0000 (oldest deadline 2024-03-30T14:34:08.216956+0000) 2024-03-30T14:34:17.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:16.728+0000 7f7898ff9700 -1 osd.7 834 heartbeat_check: no reply from 172.21.15.138:6802 osd.4 since back 2024-03-30T14:33:52.917657+0000 front 2024-03-30T14:33:52.917697+0000 (oldest deadline 2024-03-30T14:34:16.417723+0000) 2024-03-30T14:34:17.186 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:16 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7[55957]: 2024-03-30T14:34:16.728+0000 7f7898ff9700 -1 osd.7 834 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e834) 2024-03-30T14:34:18.303 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:18 smithi138 podman[77061]: 2024-03-30 14:34:18.030404296 +0000 UTC m=+5.578495148 container died 63640fde36efb93402e48ab2c9244dee8a2075e0b309c2153b8c25c191c00002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, ceph=True, io.buildah.version=1.33.5) 2024-03-30T14:34:18.686 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:18 smithi138 podman[77061]: 2024-03-30 14:34:18.306890584 +0000 UTC m=+5.854981434 container cleanup 63640fde36efb93402e48ab2c9244dee8a2075e0b309c2153b8c25c191c00002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, ceph=True, GIT_CLEAN=True, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-30T14:34:18.686 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:18 smithi138 bash[77061]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7 2024-03-30T14:34:18.686 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:18 smithi138 podman[77091]: 2024-03-30 14:34:18.549675045 +0000 UTC m=+0.516376902 container remove 63640fde36efb93402e48ab2c9244dee8a2075e0b309c2153b8c25c191c00002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:19.175 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:18 smithi138 podman[77181]: 2024-03-30 14:34:18.775555013 +0000 UTC m=+0.018136690 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:34:19.175 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:18 smithi138 podman[77181]: 2024-03-30 14:34:18.892496411 +0000 UTC m=+0.135078118 container create 696459ab5ccebc74571721178713a6ef408ec891363d6e8795bc20257f834f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:19.175 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:19 smithi138 podman[77181]: 2024-03-30 14:34:19.103463185 +0000 UTC m=+0.346044863 container init 696459ab5ccebc74571721178713a6ef408ec891363d6e8795bc20257f834f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-30T14:34:19.175 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:19 smithi138 podman[77181]: 2024-03-30 14:34:19.106545754 +0000 UTC m=+0.349127420 container start 696459ab5ccebc74571721178713a6ef408ec891363d6e8795bc20257f834f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:34:19.175 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:19 smithi138 podman[77181]: 2024-03-30 14:34:19.178411036 +0000 UTC m=+0.420992706 container attach 696459ab5ccebc74571721178713a6ef408ec891363d6e8795bc20257f834f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-30T14:34:19.436 INFO:journalctl@ceph.osd.7.smithi138.stdout:Mar 30 14:34:19 smithi138 podman[77181]: 2024-03-30 14:34:19.390473972 +0000 UTC m=+0.633055649 container died 696459ab5ccebc74571721178713a6ef408ec891363d6e8795bc20257f834f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:34:20.008 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@osd.7.service' 2024-03-30T14:34:20.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:34:20.051 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-30T14:34:20.051 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-30T14:34:20.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@rgw.foo.a 2024-03-30T14:34:20.357 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:20 smithi102 systemd[1]: Stopping Ceph rgw.foo.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:20.666 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:20 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a[64307]: 2024-03-30T14:34:20.358+0000 7f4ae5ee3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:34:20.666 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:20 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a[64307]: 2024-03-30T14:34:20.358+0000 7f4af4762ac0 -1 shutting down 2024-03-30T14:34:30.690 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:30 smithi102 bash[154105]: time="2024-03-30T14:34:30Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-30T14:34:30.690 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:30 smithi102 podman[154105]: 2024-03-30 14:34:30.375586474 +0000 UTC m=+10.126037790 container stop be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-30T14:34:30.690 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:30 smithi102 podman[154105]: 2024-03-30 14:34:30.403402883 +0000 UTC m=+10.153854198 container died be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-03-30T14:34:31.190 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:30 smithi102 podman[154105]: 2024-03-30 14:34:30.816743187 +0000 UTC m=+10.567194490 container cleanup be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-03-30T14:34:31.190 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:30 smithi102 bash[154105]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a 2024-03-30T14:34:31.190 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:30 smithi102 podman[154131]: 2024-03-30 14:34:30.984569326 +0000 UTC m=+0.598853154 container remove be068ff7f4fcd2d190fb7798910526031f420aee4e0e3f95070934aaf67c1d01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-rgw-foo-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:34:31.190 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:Mar 30 14:34:30 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-30T14:34:31.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@rgw.foo.a.service' 2024-03-30T14:34:31.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:34:31.271 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-30T14:34:31.271 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-30T14:34:31.271 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@prometheus.a 2024-03-30T14:34:31.565 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 systemd[1]: Stopping Ceph prometheus.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:31.841 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-30T14:34:31.841 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-30T14:34:31.841 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-30T14:34:31.841 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-30T14:34:31.841 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-30T14:34:31.841 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-30T14:34:31.842 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-30T14:34:31.842 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-30T14:34:31.842 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.568Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-30T14:34:31.842 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.569Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-30T14:34:31.842 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.569Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-30T14:34:31.842 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a[65849]: ts=2024-03-30T14:34:31.569Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-30T14:34:31.842 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 podman[77286]: 2024-03-30 14:34:31.578468313 +0000 UTC m=+0.116456620 container died 3ddcfe68deb6a8517311a39d97a4ae4829adac4ec061af119d9723eaa489fb8e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:32.183 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 podman[77286]: 2024-03-30 14:34:31.943263562 +0000 UTC m=+0.481251870 container cleanup 3ddcfe68deb6a8517311a39d97a4ae4829adac4ec061af119d9723eaa489fb8e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:32.183 INFO:journalctl@ceph.prometheus.a.smithi138.stdout:Mar 30 14:34:31 smithi138 bash[77286]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-prometheus-a 2024-03-30T14:34:32.309 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@prometheus.a.service' 2024-03-30T14:34:32.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T14:34:32.347 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-30T14:34:32.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm rm-cluster --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T14:34:32.531 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T14:34:34.190 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:34 smithi102 systemd[1]: Stopping Ceph alertmanager.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:34.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:34 smithi102 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a[73970]: ts=2024-03-30T14:34:34.355Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-30T14:34:34.670 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:34 smithi102 podman[154608]: 2024-03-30 14:34:34.406171842 +0000 UTC m=+0.157404973 container died 4ebdf376ccc74ca162597e215d36114ee413ded109c933cb7a859fb4a60e8010 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:34.940 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:34 smithi102 podman[154608]: 2024-03-30 14:34:34.788818736 +0000 UTC m=+0.540051858 container cleanup 4ebdf376ccc74ca162597e215d36114ee413ded109c933cb7a859fb4a60e8010 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:34.940 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:34 smithi102 bash[154608]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a 2024-03-30T14:34:35.413 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:35 smithi102 podman[154622]: 2024-03-30 14:34:35.031679639 +0000 UTC m=+0.630766474 container remove 4ebdf376ccc74ca162597e215d36114ee413ded109c933cb7a859fb4a60e8010 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:35.413 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:35 smithi102 podman[154622]: 2024-03-30 14:34:35.115732431 +0000 UTC m=+0.714819266 volume remove 4f750ed12b02f09942bc7ab9119f688621952f12c6f5fb74cfac9c0137b21da2 2024-03-30T14:34:35.413 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:35 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-03-30T14:34:35.413 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:35 smithi102 systemd[1]: Stopped Ceph alertmanager.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:35.413 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:Mar 30 14:34:35 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@alertmanager.a.service: Consumed 2.351s CPU time. 2024-03-30T14:34:35.676 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:35 smithi102 systemd[1]: Stopping Ceph node-exporter.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:35.939 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:35 smithi102 podman[154735]: 2024-03-30 14:34:35.679165853 +0000 UTC m=+0.128195202 container died 3a12e95ee124c409ff60de3f4a33196746318f51be1cff173671b17e00c35258 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:36.392 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:36 smithi102 podman[154735]: 2024-03-30 14:34:36.001891872 +0000 UTC m=+0.450921239 container cleanup 3a12e95ee124c409ff60de3f4a33196746318f51be1cff173671b17e00c35258 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:36.392 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:36 smithi102 bash[154735]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a 2024-03-30T14:34:36.392 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:36 smithi102 podman[154747]: 2024-03-30 14:34:36.211207737 +0000 UTC m=+0.532911452 container remove 3a12e95ee124c409ff60de3f4a33196746318f51be1cff173671b17e00c35258 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:34:36.392 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:36 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-30T14:34:36.690 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:36 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-30T14:34:36.690 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:36 smithi102 systemd[1]: Stopped Ceph node-exporter.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:36.690 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:Mar 30 14:34:36 smithi102 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.a.service: Consumed 4.923s CPU time. 2024-03-30T14:34:37.058 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm rm-cluster --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T14:34:37.252 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T14:34:38.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:38 smithi138 systemd[1]: Stopping Ceph iscsi.iscsi.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:39.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:38 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[60923]: debug Shutdown received 2024-03-30T14:34:49.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:48 smithi138 bash[77697]: time="2024-03-30T14:34:48Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-30T14:34:49.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:48 smithi138 podman[77697]: 2024-03-30 14:34:48.737862142 +0000 UTC m=+10.118814195 container died e61cb60d83a5f9c047d51ab3f987c138c3bdd3307bdbac736aebfea30260667d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326) 2024-03-30T14:34:49.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:49 smithi138 podman[77697]: 2024-03-30 14:34:49.090700499 +0000 UTC m=+10.471652550 container cleanup e61cb60d83a5f9c047d51ab3f987c138c3bdd3307bdbac736aebfea30260667d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-30T14:34:49.356 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:49 smithi138 bash[77697]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a 2024-03-30T14:34:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:49 smithi138 podman[77712]: 2024-03-30 14:34:49.3584306 +0000 UTC m=+0.621313301 container remove e61cb60d83a5f9c047d51ab3f987c138c3bdd3307bdbac736aebfea30260667d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:34:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:49 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-30T14:34:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:49 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-30T14:34:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:49 smithi138 systemd[1]: Stopped Ceph iscsi.iscsi.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:49.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi138.stdout:Mar 30 14:34:49 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Consumed 4.393s CPU time. 2024-03-30T14:34:51.186 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 systemd[1]: Stopping Ceph grafana.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:51.655 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=server t=2024-03-30T14:34:51.383099959Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-30T14:34:51.655 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=ticker t=2024-03-30T14:34:51.383395405Z level=info msg=stopped last_tick=2024-03-30T14:34:50Z 2024-03-30T14:34:51.655 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a[64496]: logger=tracing t=2024-03-30T14:34:51.38343955Z level=info msg="Closing tracing" 2024-03-30T14:34:51.656 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 podman[77988]: 2024-03-30 14:34:51.393485541 +0000 UTC m=+0.119004866 container died 2c0425b41c9a0ad35261b371a387b5a3230aa42caea299af9c3f1a58d2ec78ff (image=quay.io/ceph/grafana:9.4.12, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:34:51.905 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 podman[77988]: 2024-03-30 14:34:51.732408924 +0000 UTC m=+0.457928256 container cleanup 2c0425b41c9a0ad35261b371a387b5a3230aa42caea299af9c3f1a58d2ec78ff (image=quay.io/ceph/grafana:9.4.12, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:34:51.905 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 bash[77988]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a 2024-03-30T14:34:52.186 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:51 smithi138 podman[78003]: 2024-03-30 14:34:51.908601211 +0000 UTC m=+0.513333103 container remove 2c0425b41c9a0ad35261b371a387b5a3230aa42caea299af9c3f1a58d2ec78ff (image=quay.io/ceph/grafana:9.4.12, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:34:52.186 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:52 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@grafana.a.service: Deactivated successfully. 2024-03-30T14:34:52.186 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:52 smithi138 systemd[1]: Stopped Ceph grafana.a for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:52.186 INFO:journalctl@ceph.grafana.a.smithi138.stdout:Mar 30 14:34:52 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@grafana.a.service: Consumed 22.609s CPU time. 2024-03-30T14:34:52.686 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:52 smithi138 systemd[1]: Stopping Ceph node-exporter.b for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4... 2024-03-30T14:34:52.686 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:52 smithi138 podman[78117]: 2024-03-30 14:34:52.463435899 +0000 UTC m=+0.099439824 container died 560ebe94c64ed21d590fb4e4de34f76be86b20574c695331b28679bb138fb50d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:34:52.961 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:52 smithi138 podman[78117]: 2024-03-30 14:34:52.785979923 +0000 UTC m=+0.421983846 container cleanup 560ebe94c64ed21d590fb4e4de34f76be86b20574c695331b28679bb138fb50d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:34:52.961 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:52 smithi138 bash[78117]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b 2024-03-30T14:34:52.961 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:52 smithi138 podman[78131]: 2024-03-30 14:34:52.962207088 +0000 UTC m=+0.501250852 container remove 560ebe94c64ed21d590fb4e4de34f76be86b20574c695331b28679bb138fb50d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:34:52.961 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:52 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-30T14:34:53.214 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:53 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-30T14:34:53.214 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:53 smithi138 systemd[1]: Stopped Ceph node-exporter.b for 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4. 2024-03-30T14:34:53.214 INFO:journalctl@ceph.node-exporter.b.smithi138.stdout:Mar 30 14:34:53 smithi138 systemd[1]: ceph-69dd7ec8-ee9c-11ee-b647-cb9ed24678a4@node-exporter.b.service: Consumed 4.969s CPU time. 2024-03-30T14:34:53.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:34:53.951 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:34:53.980 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-30T14:34:53.982 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126/remote/smithi102/crash 2024-03-30T14:34:53.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T14:34:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T14:34:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-03-30T14:34:54.041 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126/remote/smithi138/crash 2024-03-30T14:34:54.041 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T14:34:54.083 INFO:teuthology.orchestra.run.smithi138.stderr:tar: /var/lib/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T14:34:54.083 INFO:teuthology.orchestra.run.smithi138.stderr:tar: Error is not recoverable: exiting now 2024-03-30T14:34:54.084 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-30T14:34:54.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-30T14:34:54.118 INFO:teuthology.orchestra.run.smithi102.stdout:2024-03-30T14:00:00.165657+0000 mon.a (mon.0) 1105 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-30T14:34:54.119 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-30T14:34:54.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[SEC\]' /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-30T14:34:54.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]' /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-30T14:34:54.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[WRN\]' /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-30T14:34:54.337 INFO:teuthology.orchestra.run.smithi102.stdout:2024-03-30T14:00:00.165657+0000 mon.a (mon.0) 1105 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-30T14:34:54.337 INFO:tasks.cephadm:Compressing logs... 2024-03-30T14:34:54.338 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T14:34:54.381 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T14:34:54.406 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-30T14:34:54.406 INFO:teuthology.orchestra.run.smithi102.stderr:: No such file or directory 2024-03-30T14:34:54.406 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.a.log 2024-03-30T14:34:54.406 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T14:34:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log 2024-03-30T14:34:54.407 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.a.log: /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-03-30T14:34:54.409 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-03-30T14:34:54.409 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mgr.y.log 2024-03-30T14:34:54.409 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-03-30T14:34:54.411 INFO:teuthology.orchestra.run.smithi138.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-30T14:34:54.411 INFO:teuthology.orchestra.run.smithi138.stderr:: No such file or directory 2024-03-30T14:34:54.411 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-03-30T14:34:54.411 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T14:34:54.411 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-03-30T14:34:54.411 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.6.log: /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-03-30T14:34:54.411 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-03-30T14:34:54.413 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.4.log: /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mgr.x.log 2024-03-30T14:34:54.414 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/tcmu-runner.log 2024-03-30T14:34:54.414 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-30T14:34:54.414 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/tcmu-runner.log: 67.2% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/tcmu-runner.log.gz 2024-03-30T14:34:54.414 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T14:34:54.414 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 81.5% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T14:34:54.414 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T14:34:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mgr.y.log: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T14:34:54.415 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-30T14:34:54.419 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log: 87.9% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T14:34:54.419 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.b.log 2024-03-30T14:34:54.420 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-volume.log: 92.3% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T14:34:54.421 INFO:teuthology.orchestra.run.smithi138.stderr: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T14:34:54.423 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T14:34:54.424 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mgr.x.log.gz 2024-03-30T14:34:54.426 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 89.0% 95.2% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T14:34:54.426 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-03-30T14:34:54.434 INFO:teuthology.orchestra.run.smithi138.stderr: 91.0% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T14:34:54.435 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T14:34:54.435 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.2.log: 93.3% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T14:34:54.436 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.c.log 2024-03-30T14:34:54.436 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-volume.log: -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T14:34:54.462 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T14:34:54.471 INFO:teuthology.orchestra.run.smithi102.stderr: 94.1% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log.gz 2024-03-30T14:34:54.963 INFO:teuthology.orchestra.run.smithi102.stderr: 90.8% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mgr.y.log.gz 2024-03-30T14:34:55.490 INFO:teuthology.orchestra.run.smithi138.stderr: 92.9% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.b.log.gz 2024-03-30T14:34:56.069 INFO:teuthology.orchestra.run.smithi102.stderr: 92.5% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.c.log.gz 2024-03-30T14:34:57.695 INFO:teuthology.orchestra.run.smithi102.stderr: 92.0% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-mon.a.log.gz 2024-03-30T14:35:01.992 INFO:teuthology.orchestra.run.smithi138.stderr: 93.8% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-03-30T14:35:02.047 INFO:teuthology.orchestra.run.smithi102.stderr: 93.8% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-03-30T14:35:02.245 INFO:teuthology.orchestra.run.smithi138.stderr: 93.8% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-03-30T14:35:02.289 INFO:teuthology.orchestra.run.smithi138.stderr: 93.8% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-03-30T14:35:02.594 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-03-30T14:35:02.776 INFO:teuthology.orchestra.run.smithi102.stderr: 93.9% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-03-30T14:35:02.941 INFO:teuthology.orchestra.run.smithi138.stderr: 93.9% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-03-30T14:35:02.943 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-03-30T14:35:02.943 INFO:teuthology.orchestra.run.smithi138.stderr:real 0m8.547s 2024-03-30T14:35:02.943 INFO:teuthology.orchestra.run.smithi138.stderr:user 0m31.758s 2024-03-30T14:35:02.943 INFO:teuthology.orchestra.run.smithi138.stderr:sys 0m1.165s 2024-03-30T14:35:03.677 INFO:teuthology.orchestra.run.smithi102.stderr: 93.9% -- replaced with /var/log/ceph/69dd7ec8-ee9c-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-03-30T14:35:03.679 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-30T14:35:03.679 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m9.286s 2024-03-30T14:35:03.679 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m37.587s 2024-03-30T14:35:03.679 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m1.320s 2024-03-30T14:35:03.680 INFO:tasks.cephadm:Archiving logs... 2024-03-30T14:35:03.680 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126/remote/smithi102/log 2024-03-30T14:35:03.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T14:35:08.736 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126/remote/smithi138/log 2024-03-30T14:35:08.737 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T14:35:13.384 INFO:tasks.cephadm:Removing cluster... 2024-03-30T14:35:13.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo cephadm rm-cluster --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 --force 2024-03-30T14:35:13.574 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T14:35:13.877 DEBUG:teuthology.orchestra.run.smithi138:> sudo cephadm rm-cluster --fsid 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 --force 2024-03-30T14:35:14.070 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: 69dd7ec8-ee9c-11ee-b647-cb9ed24678a4 2024-03-30T14:35:14.361 INFO:tasks.cephadm:Teardown complete 2024-03-30T14:35:14.362 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T14:35:14.372 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T14:35:14.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T14:35:14.376 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T14:35:14.454 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T14:35:14.463 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T14:35:14.463 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:35:14.465 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:35:14.477 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:35:14.479 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:35:14.482 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:35:14.482 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-30T14:35:14.482 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +92us[ +92us] +/- 65ms 2024-03-30T14:35:14.483 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 3 7 377 32 -186us[ -192us] +/- 78ms 2024-03-30T14:35:14.483 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +155us[ +148us] +/- 72ms 2024-03-30T14:35:14.483 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:35:14.484 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:35:14.484 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-03-30T14:35:14.484 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 96 +95us[ +94us] +/- 98ms 2024-03-30T14:35:14.484 INFO:teuthology.orchestra.run.smithi138.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -236us[ -236us] +/- 78ms 2024-03-30T14:35:14.484 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 33 +171us[ +170us] +/- 72ms 2024-03-30T14:35:14.484 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:35:14.484 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T14:35:14.493 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T14:35:14.493 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T14:35:14.502 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T14:35:14.600 DEBUG:teuthology.orchestra.run.smithi138:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T14:35:14.673 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T14:35:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:35:14.703 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T14:35:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:35:14.733 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T14:35:14.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T14:35:14.750 INFO:teuthology.task.internal:Duration was 3241.758824 seconds 2024-03-30T14:35:14.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T14:35:14.759 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T14:35:14.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T14:35:14.761 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T14:35:14.801 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:35:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:35:15.231 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T14:35:15.232 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-03-30T14:35:15.232 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T14:35:15.292 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-03-30T14:35:15.292 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T14:35:15.321 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T14:35:15.321 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T14:35:15.335 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T14:35:15.390 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T14:35:15.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T14:35:15.404 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T14:35:15.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T14:35:15.856 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T14:35:15.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T14:35:15.882 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T14:35:15.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T14:35:15.918 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:35:15.925 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:35:15.948 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-03-30T14:35:15.975 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-03-30T14:35:16.027 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:35:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:35:16.041 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T14:35:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:35:16.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T14:35:16.091 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T14:35:16.092 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126/remote/smithi102 2024-03-30T14:35:16.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T14:35:16.154 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126/remote/smithi138 2024-03-30T14:35:16.155 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T14:35:16.192 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T14:35:16.193 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T14:35:16.195 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T14:35:16.241 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T14:35:16.254 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T14:35:16.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T14:35:16.264 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T14:35:16.264 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T14:35:16.266 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T14:35:16.278 INFO:teuthology.orchestra.run.smithi102.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 14:35 /home/ubuntu/cephtest 2024-03-30T14:35:16.296 INFO:teuthology.orchestra.run.smithi138.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 14:35 /home/ubuntu/cephtest 2024-03-30T14:35:16.297 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T14:35:16.306 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T14:35:16.325 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T14:35:16.349 DEBUG:teuthology.nuke:shortname: smithi102 2024-03-30T14:35:16.349 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:35:16.371 DEBUG:teuthology.nuke:shortname: smithi138 2024-03-30T14:35:16.371 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:35:16.374 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:31:26.845630', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmVzuHzB7tOyXUtvJQhmRnarhqnL7kpm8q5+RZnh9egEBaAopWK4pdCNeD8FPLU9PKFuTLSirtTGHU3PSSONdA='} 2024-03-30T14:35:16.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 13:31:26.847078', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFIO3nS3DhsYqCK/Ntvgz3OpZ5cEeUTG/bEzhm96Uc4sb/TYQEwp5hc8N1hlG3RHk8vVqMdeRnIzHFb4IhAM9pk='} 2024-03-30T14:35:16.423 INFO:teuthology.orchestra.console.smithi102:Power off 2024-03-30T14:35:16.423 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T14:35:16.437 INFO:teuthology.orchestra.console.smithi138:Power off 2024-03-30T14:35:16.437 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T14:35:16.448 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-03-30T14:35:16.449 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:16.459 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off 2024-03-30T14:35:16.459 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:16.472 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-30T14:35:16.482 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-30T14:35:20.474 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:20.483 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:20.583 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-30T14:35:20.591 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-30T14:35:24.585 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:24.594 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:24.696 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-30T14:35:24.703 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-30T14:35:28.701 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:28.709 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T14:35:28.812 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-03-30T14:35:28.818 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-03-30T14:35:28.913 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-03-30T14:35:28.919 INFO:teuthology.orchestra.console.smithi138:Power off completed 2024-03-30T14:35:29.023 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} duration: 3241.758823633194 failure_reason: '"2024-03-30T14:00:00.165657+0000 mon.a (mon.0) 1105 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-30T14:35:29.023 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T14:35:29.124 INFO:teuthology.run:FAIL